META_ITF_LOCK_index should not be used to flag the status of the interferometer with second precision. Metatron runs is a slow process, with approximately 1 second long refresh time (but not constant). META_ITF_LOCK_index reports when Metatron learned that the interferometer has unlocked, which will always be 1 or 2 seconds after the fact.

Better channels to use to learn quickly that the ITF has unlocked are

• LSC_ENABLE - this channels is equal to 1 the interferometre is controlled, and drops to 0 when the real time control gives up on controlling the interferometer (switches off the feedback to the mirrors) after an unlock
• SDB1_fast_shutter_sw - this channels is at 1 when the shutter blocks the beam at the output of the interferometer and at 0 when the shutter is open, so whenever it is at 1 for certain there is no GW signal to look at. In some cases it is an earlier warning of the unlock (by a fraction of a second)

See figure 1 for an example.

As Michal said, the Metatron latency due to the need to pass on information at 1 Hz in between nodes is well-known. This is dealt with offline by the CAT1 flag V1:DQ_SCIENCE_FALSE that checks for inconsistencies between ITF_LOCK and ITF_Mode == 1 (SCIENCE) -- and also by V1:DQ_LOCK_LAST_05S that removes the last 5 seconds of each lock segment. I'll double-check tomorrow if adding the min values of LSC_ENABLE and SDB1_fast_shutter_sw removes more data.

That being said, I don't quite understand the problem reported by MBTA: from https://logbook.virgo-gw.eu/virgo/uploads/47687_1573925680_Unlock-Nov14-12h54.png, the bit 0 of the state vector goes down at 12:54:18 UTC, meaning that from that time onwards, h(t) is not correctly computed. So these data are bad. Pipelines should use data when bits 0, 1 and 10 are active.

The issue that Benoit reports is that h(t) is corrupted 4 seconds before the bit 0 of the state vector goes down 12:54:18 UTC, h(t) is already bad at 12:54:14 UTC. The reason is that h(t) is not a time domain code, but a frequency domain one done in blocks of 8 seconds. If any data input data in that 8 seconds is not good, then all 8 seconds of h(t) are corrupted. So h(t) is not only bad after the unlock, but also before the unlock, because it doesn't know about the unlock. There are two solutions:

• Use better information in Hrec_hoft for knowing when an unlock happens. Hrec can't use the state vector as that is computed further down the DAQ chain. So probably additional checks would need to be implemented into Hrec.
• Acknowledge that Hrec can be corrupted in the last 8 seconds before an unlock, and remove the last 8 seconds instead of 5 seconds before an unlock using a CAT1 flag that is accessible to online pipelines (such as MBTA).

Triggered by this thread and additional e-mail exchanges, I've improved the script used to find lock losses to define offline segments and vetoes. The new version combines information from Metatron and from fast signals; it also removes the last 10 seconds before each identified delock, in order to avoid any corrupted h(t) sample -- the previous version was only trimming away 5 seconds / delock. A new CAT1 flag covering the whole O3a has been generated and uploaded to DQSEGDB. CBC and burst veto definer files will be updated accordingly. The new flag adds about 36 minutes of CAT1 time (very small amount w.r.t. O3a) without increasing much the number of SCIENCE segments available for offline analysis as it can gather short CAT1 segments that were 'precursors' of the delock. These additional CAT1-flagged data were anyway in too short segments to be of any use for offline data analysis.

This new flag will be used by default during O3b.

My understanding is that Hrec will work on identifying better delocks to improve online CAT1 flags.

Today, around 14:15 UTC, while ITF was unlocked, I have restarted the Hrec processes with version v3r09p5.
This version of Hrec has a new usage of the LOCK_CHANNEL keyword , which allows to use both
channels META_ITF_LOCK_index and SDB2_B1_DC to say that ITF is locked or not.
The use of SDB2_B1_DC will allow shorter latency and should prevent h(t) to be computed when it should not.
Plot 1 shows one example of unlock, with the h(t) signal affected by the too late detection of unlock using only META_ITF_LOCK_index
and the corresponding state vector late change.
Plot 2 shows for the same unlock, the h(t) correctly put to zero when we use also SDB2_B1_DC > 0.1 condition
and the corresponding state vector correct change.