Reports 1-1 of 1 Clear search Modify search
AdV-COM (automation)
bersanetti, carbognani - 20:06 Wednesday 03 April 2024 (63828) Print this report
Introduction of the ITF_CONDITIONS node and update of the ITF_STATUS node

Today we put in operation a new node, ITF_CONDITIONS, and we updated the already existing ITF_STATUS one. The underlying process is described in gitlab issue #19, and the main goal is to decouple what we call ITF Modes (Commissioning, Calibration, Maintenance, etc.) to ITF Conditions (Not Locked, Locking, Locked) as conditions are something the machine is doing whatever it is we want to do with it. The Conditions are:

ITF Condition META_ITF_CONDITIONS_index
Not Locked 10
Locking 20
Llocked 30

Conditions are automatically generated by the ITF_CONDITIONS node, which only has a few states selectable; the main is DOWN, which is basically the default state, then the node will transition automatically to the three main conditions, dependently of the status of the interferometer itself, by tracking the index of the main ITF_LOCK node. The only other functionality ITF_CONDITIONS has is the Autorelock Failsafe one, which has been migrated from ITF_STATUS. This is working exactly as before, by selecting the state of the same name. The STOP state will exit from this.

Being stripped of such functionality and old unnecessary O3 code, now ITF_STATUS only deals with the modes, that have to be set manually by people, and which describe what the machine is targeted to do (or what is preventing the same thing); the modes are changed with respect to the past, and we have a few additions; each corresponds to a given node state as usual. The complete list is in the following table, and the number represents the value of the DQ_META_ITF_Mode channel (the new modes are listed in bold):

ITF Mode DQ_META_ITF_Mode value
Science 1
Adjusting -1
Commissioning -3
Maintenance -4
Calibration -5
Injection -6
PrepareScience -7
Earthquake -8
BadWeather -9
Upgrading -11
Troubleshooting -12
Dqstudies -13

About the new modes: Earthquake and BadWeather are to be set when the machine is in some kind of trouble in locking, because we have the effects of one of the two, the second one (BadWeather) referring mostly to high wind and/or high microseismic activity.

PrepareScience is the default mode when none of the others is actually happening: this means thatif  we are locking only to reach the nominal state and then take data, this is the mode. As usual, the transition to Science can be done from any state, and the node itself will take care of checking whether this is legit or not, based on the ITF_LOCK index as usual; the transition will be not even started in the case, avoiding going in and out even for a second. If for any reason we exit from Science (unlock, transition between LOW_NOISE_3 to LOW_NOISE_3_SQZ, etc), the node will automatically fall to PREPARE_SCIENCE, losing memory of the previous state, which we think is the right thing to do. After that, the mode will stay PREPARE_SCIENCE waiting for a manual operation.

To recap: Modes and Conditions are decoupled, independent information. The information Not Locked, Locking and Locked now are the (only) Conditions, and whatever was looking at them based on the DQ_META_ITF_Mode value should move to check the ITF_CONDITIONS_index instead.

The Modes are instead almost identical to the previous behaviour, and we just have three new modes.

The node graphs of the two nodes are attached.


EDIT: we temporarily reverted to the old ways, as there is some unforeseen and currently unexplained bug impacting the production of h(t), which should only look at META_ITF_LOCK_index, which wasn't affected by the aforementioned changes.

Non-image files attached to this report
Comments to this report:
bersanetti, carbognani, masserot, mours - 22:37 Friday 05 April 2024 (63860) Print this report
This afternoon we actually put in operation this update.
The problem we were facing with the introduction of the new ITF_CONDITIONS node was that we had 2 nodes using the DQ_META SER prefix and this was creating conflicts leading to the unavailability of some channels including META_ITF_LOCK..index.
The solution proposed by Benoit was to replace the line:
FDOUT_CM FbmAlp "#SER V1:META_* V1:DQ_*" 0 -1

in the automation node config by
FDOUT_SELECT_CHANNELS "V1:META_* V1:DQ_*"
FDOUT_CONVERT_SERDATA
FDOUT_CM FbmAlp "*" 0 -1

so to convert the SerData to AdcData at the output of the Metatron node and avoid the conflict.
We tried the suggested conversion to AdcData but it was not working initially. Benoit diagnosed that the issue may have been associated to the version of Fd being used by metatron.
Indeed by creating a dev version of metatron (/virgoDev/metatron/v0r3p1) running into a conda env updated to Fd 8.61.1 (/virgoDev/mamba/env/automation) and by just applying the conversion to AdcData for the ITF_CONDITIONS node things were then working fine.

So the channels now created by ITF_LOCK (in particular META_ITF_LOCK..index) and ITF_STATUS are as before (SerData) and only DQ_META_AUTORELOCK_*, DQ_META_ITF_LOCKED, DQ_META_NOMINAL_LOCK, and DQ_META_ITF_LOCK_REQUEST are now AdcData.

We keep things running in the current configuration further checking for any other unintended side effects. Plotting of the most relevant channels looks ok (Fig. 1) and the ITF Mode could be driven from the updated ITF_STATUS node without problem.



Images attached to this comment
carbognani - 17:12 Tuesday 09 April 2024 (63906) Print this report

As one outcome of the dedicated discussions, the DQ_META_ITF_Mode coding has been updated so that indices of modes which do not exist anymore are not reused, this lead to:

  • Science = ["Science", 1]
  • Adjusting = ["Adjusting", -1]
  • Commissioning = ["Commissioning", -3]
  • Maintenance = ["Maintenance", -4]
  • Calibration = ["Calibration", -5]
  • Injection = ["Injection", -6]
  • Upgrading = ["Upgrading", -11]
  • Troubleshooting = ["Troubleshooting", -12]
  • Dqstudies = ["Dqstudies", -13]
  • PrepareScience = ["Prepare_Science", -14]
  • Earthquake = ["Earthquake", -15]
  • BadWeather = ["Bad_Weather", -16]


Every application looking at DQ_META_ITF_Mode should check and eventually adapt to the new coding.

masserot, viret - 20:45 Thursday 19 February 2026 (68720) Print this report

The side issue has been fixed by restarting the Fd dataDisplay providers:  Fbm{Det, Main, Moni, Susp}Users, FbmPyAccess and FbmStDy{,Bis} servers.

The problem seems related to the channel list built by these servers and used by the DataDisplay.

As already reported, there weren't duplicated channels in the DAQ streams

bersanetti, carbognani - 1:03 Friday 20 February 2026 (68719) Print this report

In order to align the metatron instance running in Cascina we have removed the dev version of metatron (/virgoDev/metatron/v0r3p1) used to run ITF_CONDITIONS node.

We have then promoted the conda env updated to Fd 8.61.1 (/virgoDev/mamba/env/automation) into production under /virgo/conda/envs and then restarted the same version of metatron (/virgoApp/metatron/v0r3p0) for all automation nodes on top of the updated conda environment.

We first tested only the ITF_CONDITIONS and ITF_LOCK nodes, in order to verify that all channels supposed to go in the DAQ actually do, as this was the showstopper reported in v68573. After verifying this, we stopped and restarted all the nodes with the new underlying conda environment, without changing the version of metatron itself.

We tested a full lock acquisition up to LOW_NOISE_3_ALIGNED, with no issues.


A side issue we found before starting the activity is the presence of a few duplicated channels, only on the online data and not the stored one. These are the channels which are pushed to the DAQ by ITF_CONDITIONS (DQ_META_AUTORELOCK_*, DQ_META_ITF_LOCK_REQUEST and DQ_META_NOMINAL_LOCK). As they don't appear duplicated in the stored data, we don't know if this behaviour started today after a couple of tests with the previous metatron configuration, or it was already the case since some time. We'll try to figure this out offline.

bersanetti - 1:06 Friday 20 February 2026 (68721) Print this report

Tonight, after each unlock, ITF_CONDITIONS got stucked in NOT_LOCKED, without transitioning again to LOCKING, therefore blocking ITF_LOCK in DOWN.

Tomorrow we will investigate this unexpected behaviour and the link to what was done earlier today. For tonight, I disabled the autorelock failsafe and let the ITF start locking again.

Search Help
×

Warning

Error

The present report has been modified outside this window. Please check for its integrity in the main page.

Refreshing this page will move this report into drafts.

×