Today I continued to work on the node:

• the automatic transition to Science has been set after 180 s of LOCKED state; this can be updated and is configurable inside the ITF_STATUS.ini file;
• improved the engagement of AutoScience: if requested, it will be engaged in any state of the interferometer, not only if just LOCKED;
• improved the node management between the two features AutoRelock and AutoScience;
• added a new channel, DQ_META_AUTORELOCK_FAILED, which will be set to 1 when the Failsafe triggers and we stop trying to lock; this should help in setting up flags and alarms on the DMS.

Still missing:

• test of unlock when the feature is activated;
• squeezing management and multiple-crossing of Science-viable states;
• implement a possible reason to trigger an intervention, which is maybe better to set up directly in the DMS: in case we consistently lock, but the locks last a short time, there is no way now to monitor this, as the AutoRelock will be reset at each successful acquisition, while AutoScience has no ways of failing as it is just a flag set once we're locked; maybe some check made on the duration of DQ_META_ITF_Mode being 1 for how long can be implemented in the DMS:
• still in the DMS, make explicit that the AutoScience is enabled.

The feature has been turned on and it will be in operation for the weekend, supervised by the operators to spot any misbehaviour. It can be deactivated by asking the STOP state to ITF_CONDITIONS, in case of Calibration activities and of any manual intervention.

The tests done during the weekend have been mostly successful, with the only persistent issue being that ITF_LOCK is requested LOW_NOISE_1 instead of LOW_NOISE_3, after an unlock.

I think I figured out that this is due to the fact that when a node gets managed (as ITF_LOCK does while being under AutoScience) it remembers the state requests of its managers, overriding the ones done manually on the interface. LOW_NOISE_1 is the last request made by the CALI node during the calibration shift last Monday, and even if ITF_CONDITIONS redundantly asks LOW_NOISE_3 when we are in LOW_NOISE_3 just before Science, apparently both requests are still valid.

Here an extract of one of the ITF_LOCK log files, where we can also see that the ITF_CONDITIONS node runs as virgorun and not virgod:

2024-12-14T17:58:45.504Z USERINPUT REQUEST: olserver52.virgo.infn.it virgod  old=LOW_NOISE_3 new=LOW_NOISE_1 [leftover request done by CALI]
2024-12-14T17:58:46.487Z USERINPUT REQUEST: olserver52.virgo.infn.it virgorun  old=LOW_NOISE_1 new=LOW_NOISE_1 [request by ITF_CONDITIONS, none at the time]
2024-12-14T18:03:01.119Z USERINPUT REQUEST: ctrl24.virgo.infn.it amagazzu  old=LOW_NOISE_1 new=LOW_NOISE_3
2024-12-14T18:49:42.613Z USERINPUT REQUEST: olserver52.virgo.infn.it virgorun  old=LOW_NOISE_3 new=LOW_NOISE_3 [request by ITF_CONDITIONS just before Science mode]
2024-12-14T19:31:20.373Z USERINPUT MODE: olserver52.virgo.infn.it virgorun  old=1 new=0
2024-12-14T19:32:08.399Z USERINPUT MANAGER: olserver52.virgo.infn.it virgod  old= new=CALI ['hanging' management by CALI]
2024-12-14T20:07:30.138Z USERINPUT OP: ctrl24.virgo.infn.it amagazzu  old=2 new=2
2024-12-14T20:07:34.653Z USERINPUT MODE: ctrl24.virgo.infn.it amagazzu  old=1 new=0
2024-12-14T20:12:38.527Z USERINPUT MANAGER: olserver52.virgo.infn.it virgorun  old= new=ITF_CONDITIONS
2024-12-14T20:12:40.366Z USERINPUT MANAGER: olserver52.virgo.infn.it virgorun  old=ITF_CONDITIONS new=ITF_CONDITIONS
2024-12-14T20:15:42.484Z USERINPUT REQUEST: olserver52.virgo.infn.it virgorun  old=LOW_NOISE_3 new=LOW_NOISE_3
2024-12-16T07:06:13.652Z USERINPUT REQUEST: olserver52.virgo.infn.it virgod  old=LOW_NOISE_3 new=LOW_NOISE_1
2024-12-16T07:39:08.277Z USERINPUT REQUEST: ctrl24.virgo.infn.it fabiog  old=LOW_NOISE_1 new=LOW_NOISE_3

I think that this is because we never put much attention collectively on how to set nodes free when management is not needed anymore, i.e. setting AUTO on the managee's interface apparently is not enough; maybe writing the internal channel ezca['GRD-ITF_LOCK_MODE'] = 'AUTO' is enough, but probably it is best to use the Metatron call node.release() (which should be the opposite of nodes.set_managed()).

More testing will be done on this, other pending items (SQZ and DMS) to be checked during the week.

Yesterday the work continued, on two different topics:

• we implemented the nodes.release() instruction in both the CALI and ITF_CONDITIONS nodes, then we forced a DOWN->INIT->DOWN cycle on CALI in order to properly detach ITF_LOCK from it; then we operated ITF_CONDITIONS normally. When we unlocked because of the earthquake there was no state request done by ITF_LOCK, so apparently the only one active was the one issued by ITF_CONDITIONS, this should mean that the issue is solved;
• some update is needed for the ITF_EVENTS node, in order to properly display on the DMS and the ITF Operations page what is going on and if AutoRelock/AutoScience are enabled; because the node cannot read from the DAQ channels put there by the automation itself, we decided to make ITF_CONDITIONS write somewhere whether the two functionalities are active, so that ITF_EVENTS could read from there; for this reason I decided to give ITF_CONDITIONS the dignity of its own configuration file (ITF_CONDITIONS.ini) instead of using the one of ITF_STATUS, at the cost of a slight increase in duplication, as now all three "detector" nodes (_STATUS, _CONDITIONS and _EVENTS) have in their configuration files the parameters of the two Science-viable states; this duplication is however small and those parameters will hardly change for a long time.

The ITF_CONDITIONS.ini file is the one driving both AutoRelock and AutoScience (in principle the last four parameters of ITF_STATUS.ini could now be removed); they both have a section, as follows:

[AUTORELOCK]
max_lock_retries = 5
max_trans_duration = 3600
itf_lock_fallback_state = LOCKED_ARMS_IR
autorelock_enabled = False

[AUTOSCIENCE]
science_timer = 180
autoscience_enabled = False

The first section has the usual parameters for the AutoRelock Failsafe (number of attempts, maximum duration of a single attempt, fallback state in case of failure), and the second section only contains the timer after which we automatically set up the Science state.

The two _enabled boolean parameters are the two values written by the ITF_CONDITIONS node, they are not to be changed by hand: they are the ones needed to be read from ITF_EVENTS as explained above.

Today I improved the STALL detection from ITF_CONDITIONS towards ITF_LOCK and ITF_STATUS. I added to the node, only in the main() method in DOWN, the same code which is widely used in the unstall_nodes decorator, without using the decorator itself to avoid issues when ITF_CONDITIONS is not managing the two.

After trying to unstall both nodes, what is done is asking both nodes the states that they jumped to on their own in occasion of an unlock (so DOWN and NOT_LOCKED respectively); at this point it is needed (in the run() method) to ask again the target state for ITF_LOCK, which will be the last Science-allowed state it was into. For this reason, just before putting Science mode, the ITF_CONDITIONS node also saves in the configuration file the name of such state (in the same way as the two boolean flags described in a previous comment), so that once in DOWN such state can be loaded from the configuration file and asked again to restart the lock acquisition, hopefully getting rid of the stalling issues observed a few nights ago.

The ITF_CONDITIONS node has been reloaded but I will need a couple of full lock acquisition cycles to understand if everything is fixed now or if additional debug is needed.