I had a look at  the unlocks after the LSC_Acl mimimun elapsed_time tuning performed the 2025-06-24-16h20m30s-UTC:

• 2025-06-24-17h35m14s-UTC  - Sc_WE glitch
• 2025-06-24-19h05m24s-UTC - Sc_WE glitch
• 2025-06-24-19h37m27s-UTC - Sc_WE glitch
• 2025-06-24-22h09m14s-UTC - Sc_WE glitch
• 2025-06-24-22h37m44s-UTC -  unknown reason : not the standard shape of DSP glitches
• 2025-06-24-23h07m36s-UTC - Sc_WE glitch
• 2025-06-26-00h11m41s-UTC - Sc_WE glitch
• 2025-06-26-03h57m50s-UTC - Sc_WE glitch
• 2025-06-26-04h30m26s-UTC - Sc_WE glitch
• 2025-05-26-05h05m48s-UTC - Sc_WE glitch
• 2025-05-26-05h16m06s-UTC - unknown reason : not the standard shape of DSP glitches

Remarks

• None error in the return channels to the SUSP_rtpc
• None unlock due to NE DSP glitches
• the majority of the unlocks are due to WE DSP glitches

As new trial, the LSC_Acl minimum elapsed_time has been set to 33us instead of 29us previously at 2025-06-25-05h29m20-UTC

After the latest unlock, a few probes have been added from Sc_WE, in order to monitor more GLB input channels. The signals have been added to existing unused channels and the name of the probes have not been changed. The legend is:

Sc_WE_noise --> GLB00 (LOCK FLAG)

Sc_WE_txAA --> GLB0a (calib lines on MI)

Sc_WE_tyAA --> GLB0b (calib lines on MA)

Sc_WE_ErrPost --> GLB0c (calib lines on MI)

The signals from calib are added to the locking correction (MA or MI) before the latest high pass filter, compensating the analog filter after the DACs.

A glitch has been catched; all the new probes show an anomalous sample. In one of the signals, the bad data is large enough to explain the unlock.

As a side effect of the download of the master board, a few probes don't work anymore. They are the output signals of MIR_PSDF, the signals acquired by the board itself. The problem seems to be only on the probes, because the signals are used to generate MIR_Z, MIR_TX and MIR_TY, which don't show any issue. They are anyway signals not used for the control.

It is hard to imagine how a glitch at ~400 can be created out of a calibration noise channel that usually contains only small numbers. One possibility is if the issue is that the channel order become wrong for a few samples. So that the LSC_B8_DC (third channel in the packet) is received instead of LSC_CAL_WE_MIR_Z_NOISE (thirteenth channel in the packet). LSC_B8_DC has a mean value of ~535, so that could create a value around 400, if it is wrong for 3 out of 4 of the 40kHz samples used by the DSP and recorded back in the DAQ at 10kHz.

Another event with similar values, in different position. The channel with the power could be probably removed, or at least reduced in amplitude before sending it to the DSP. I would also propose to remove the unused channels: MAR calib and a channel supposed to send some damping of violin modes, never used.

For tracing the WE DSP issue,  (see the plot)

• Sc_WE_noise  is equal to LSC_WE_LOCK_FLAG delayed
• Sc_WE_txAA is equal to CAL_WE_MIR_Z_CORR delayed
• Sc_WE_tyAA  is equal to CAL_WE_MAR_Z_CORR delayed
• Sc_WE_ErrPost is equal to  CAL_WE_MIR_Z_NOISE delayed

In operation since 2025-06-26-12h03m19-UTC

As suggested by Michal: 2 packets mixed at the DSP board level

• event 2025-06-26-12h46m53s-UTC
• event 2025-06-25-17h06m11s-UTC

The removal of two GLB channels have been performed this morning, both on NE and WE. The place of calibration lines in the list of the channels has been anticipated: now the last two are angular corrections, not used for standard operation ad temporarily disabled putting zero in the DSP gain. Those channels ar not visible in WE data; there are still two calibration channels (MAR has been rmoved). The probe remained free (Sc_WE_tyAA) is now used for the GLB channel corresponding to B8_DC.

So far, no glitch has been detected. Also LOCK FLAG, which uses to show a little distorsion when a glitch occurs, seems totally clean.

Since the operations of this morming  performed around 2025-06-27-09h29m12-UTC,  the Sc_WE debugging channels are  (see the plot)

• Sc_WE_noise  is equal to LSC_WE_LOCK_FLAG delayed
• Sc_WE_MIR_Z_CORR is equal to LSC_WE_CORR_SHAPED delayed
• Sc_WE_txAA  is equal to CAL_WE_MIR_Z_CORR delayed
• Sc_WE_tyAA is equal to LSC_B8_DC delayed
• Sc_Err_post is equal to  CAL_WE_MIR_Z_NOISE

Two more glitches have been detected on WE, after one quiet day. The first, at 14 UTC (1435154659), looks very similar to the others: on one CAL channel, the third to last of the packet, we have 0.8. On LOCK FLAG we have the usual small glitch. The event occurred when the ITF was not locked; in case it was, the glitch was potentially destructive.

The second event, at 16_30 UTC (1435163616), occurred in observing mode and had no effect: all the visible channels show only a small distorsion, like a missing sample. Maybe the wrong data were present only in the last two channels, not visible and not important.

One more WE glitch, yesterday at 22 UTC (1435184344), which unlocked the ITF. The event rate appears to have decreased significantly, but the issue is still present, An attempt to improve the patch could be done, anticipating more the sensible channels.

Below the list of WE DSP glitches found  without unlock of the ITF

• 2025-06-28-16h33m18s-UTC
• 2025-06-28-22h18m46s-UTC
• 2025-06-29-20h26m56-UTC: all the channels filled with zero without any warning or error message reported by ISC_Fb
• 2025-06-29-23h18m54s-UTC
• 2025-06-30-11h33m10s-UTC
• 2025-06-30-15h46m47s-UTC

After the WEb UPS intervention, the Sc_WE glitch debugging channels have changed, meaning that the DSP codes are not exactly the same as before the UPS intervention

• before 2025-07-01-05h00m00s-UTC
• after 2025-07-01-16h00m00s-UTC
  • Sc_WE_txAA is not CAL_WE_MIR_Z_CORR delayed
  • Sc_WE_ErrPost is not CAL_WE_MIR_Z_NOISE delayed