At my arrival, locking acquisition was in progress.

• 13:17UTC, ITF in Science
• 15:40UTC, Calibration mode to start the planned activity
  • 15:41UTC, ISC longitudinal noise injections (inject_lsc.py)
  • 15:56UTC, ITF unlocked, so I started with the other measurements planned with ITF unlocked
  • 15:58UTC, Measurement of the TF between CAL and Sc channels, and of PCal sensing delays
  • 16:19UTC, Measurement of actuators response for PR and BS mirrors
  • 16:53UTC, calibration failed due to an unlock of the injection
  • 16:57UTC, Troubleshooting mode to allow M. Gosselin and P. Spinicelli to realign the BPC
  • 18:02UTC, BPC alignment concluded and locking acquisition started
  • 18:48UTC, Troubleshooting mode to allow D.Bersanetti to restore the locking acquisition
  • 19:51UTC, back to Science

ISC On-Call
18:30UTC, ITF unlocked five times at Acquire DRMI step or before (first when I disaligned the PR to align the CITF, see 1st plot), so I called D.Bersanetti to check the status

VPM
16:49UTC, "FmRds_II" process restarted after an unexpected loss of connection

Guard Tours (UTC):
17:50 - 18:30
20:55 - 

Below the list of WE DSP glitches found  without unlock of the ITF

• 2025-06-28-16h33m18s-UTC
• 2025-06-28-22h18m46s-UTC
• 2025-06-29-20h26m56-UTC: all the channels filled with zero without any warning or error message reported by ISC_Fb
• 2025-06-29-23h18m54s-UTC
• 2025-06-30-11h33m10s-UTC
• 2025-06-30-15h46m47s-UTC

ITF found locked at LN3. SCIENCE mode.

At 06:24 UTC I set ITF in ADJUSTING mode for a extraordinary nitrogen refill at WE. Unfortunately ITF unlocked during the truck transit. 
Relocked at 08:13 UTC at LN3 after seferal attempts. SCIENCE mode set. AUTOSCIENCE engaged.

12:08 UTC, ITF unlocked (TBC). Relocking in progress...

ISC
06:35 UTC, under request of Mantovani, we proceeded with a step of NI Etalon up to 20.24 in 8 hours.

As a side observation here you can find a trend of the DIFFp TY gain servo on Friday (Figure 1) and on Saturday (Figure 2).

On Saturday the INJ system unlocked two times. The first unlock (around 7h UTC) brought the SIB1 bench far away from its initial position (Fig. 1) and I manually realigned it. The relock was not smooth, as the PMC unlocked and I had to discharge the corrections of the BPC card as well. After that the system relocked and the metatron INJ node worked properly. The second unlock a similar situation happened, but after bringing the bench to the nominal position, the IMC couldn't unlock. After some investigation I noticed that the  TX and TY BPC set points on the DPS cards were different from the actual values of these signals when the INJ is in golden state. So I changed the TX set point from 7 to 13 and TY from 20 to 55. I also discharged the corrections and then the system went back to its normal behaviour (Fig. 2). 

The following report has been submitted to the On-call interface.

On-call events -> Interferometer Sensing & Control

Title: DIFFp TY oscillations

Author(s): casanueva

Details:

The operator contacted me because there were few failed lock acquisitions in CARM NULL 3F, and told me he saw an oscillation on the DIFFp loops. I checked and indeed the DIFFp TY was oscillation at VERY low frequency. On Friday, I had to decrease the gain of the same loop because it was oscillation during the automated increase of gain before engaging the Full Bandwidth loop. So I had commented it. However, this was not the case anymore on Saturday night, and without the increase of gain the loop oscillated with low stability margin. So I commented back the increase of gain and it worked.

I also noticed that the servo of the DIFFp TY then increased its gain further from 8 to 40. I remember having this problem last year so I looked at old logbook entries and I found out that tuning the CH we managed to reduce this effect (#63623). To keep this in mind if it causes more problems.

* Note that any files attached to this report are available in the On-call interface.

Updated plot, using one week more of data (until this morning 2025/06/30 0800 UTC).

Same plot, updated with one week more of data.

ITF found in LOW_NOISE_3 and in Science Mode.
IT unlocked twice during the shift (22:15 UTC, 3:10 UTC), in both cases it possible to relock at the first attempt.
ITF left in Science Mode. 

Guard Tour (UTC)
23:33 - 0:13
3:04 - 3:45

ITF remained locked in LOW_NOISE_3 with SCIENCE mode (Autoscience ON) for the whole shift; BNS Range ~52 Mpc.

Guard tour (UTC)
14:15 -> 14:53
16:42 -> 17:21
20:12 -> 20:52

DAQ
FCEM, FCEB and FCIM DAQ boxes' temperature remained just above respective red thresholds (~50 degC) for almost the entire duration of shift.

The point heating with IPATSiA has changed slightly the position of the HOM in the FSR.

Figure 1 shows the order 2 peaks, these move up in frequency slightly, add additional peaks at higher frequency either appear or become more prominent

Figure 2 shows the order 6 peaks, these move down slightly, and additional peaks at lower frequency either appear or become more promintent .

Figure 3 and 4 shows the same as an ASD, comparing the time with IPATSiA off (purple) and on (blue).

The fact that additional peaks appear can make sense, adding a heatpoint will make the mirror astigmatic, as the heatpoint is not perfectly centered. 

The fact that the order 2 and 6 modes move in opposite directions is surprising.  The same had happened when the ring heater power was changed https://logbook.virgo-gw.eu/virgo/?r=67078 . I do not have an explanation for this.

One more WE glitch, yesterday at 22 UTC (1435184344), which unlocked the ITF. The event rate appears to have decreased significantly, but the issue is still present, An attempt to improve the patch could be done, anticipating more the sensible channels.

ITF found locked at LOW_NOISE_3 in SCIENCE mode.

07:51 UTC - S250629ae
14:13 UTC - S250629bs

At 06:45 UTC ITF unlocked due to the unlock of IMC. Autorelocked at first attempt. SCIENCE mode.
At 11:33 UTC ITF unlocked again due to the unlock of IMC. DET_MAIN Metatron node got stuck in unknown state. Properly recovered putting it in Init state. Relocked at first attempt. SCIENCE mode.

Guard tours (UTC):
05:00 - 05:42
08:00 - 08:40
09:10 - 09:59
12:10 - 12:53

Vacuum
(29-06-2025 03:18 - 29-06-2025 05:10) On site
Status: Ended
Description: check the level of nytrogen at WE

This night, around 2 AM local time, the relief valve of the WE external LN2 tank opened due to the tank pressure increasing to about about 3 bar. The tank LN2 content was then released. The event is confirmed by the safety valve that was found still open in discharge mode and by the analog gauge on the tank, showing about 3 bar.

The event was not traced by the remotized gauge, apparently below 2 bar, reason to be checked.

The overpressure event is linked to a problem with the tank poor insulation, which causes the inner pressure increasing. This morning we partially fixed the issue by installing a vacuum pump to improve the insulation vacuum: the insulation vacuum level is now back to 1 mbar. The pump must stay always ON and kept monitored. We keep the system running with the remaining LN2 and have asked the supply company for an emergency refill as soon as possible, normally tomorrow morning. We will coordinate with the control room for this operation.

ITF found in Trobleshooting mode; the IMC issue was just resolved and lock acquisition was just started.

The ITF reached Science mode at 21:47 UTC but it unlocked at 22:18 UTC. At the next relock I realized that the fmoderr was skipped (fmoderr_skip = True in the ITF_LOCK.ini file); I reactivated it.

Then the ITF collected multiple unlocks at CARM_NULL_1F because of DIFFp loop oscillation; at 23:10 UTC I set Troubleshooting mode and I contacted the ISC on-call (Julia). She was able to fix the problem, the ITF survived to CARM_NULL_1F and Science mode was set at 0:09 UTC.

The ITF unlocked at 2:14 UTC, Science mode set at 2:56 UTC.

Vacuum

At 3:00 UTC the guardian reported that the vacuum tank at WE was leaking nitrogen; on the DMS the problem was hidden by by a flag that was already red. I contacted the Vacuum on-call; at around 4:00 UTC he came on site and he performed and he verfied thestatus of the plant; the problem will be fixed tomoroow.

Guard tours (time in UTC)

23:00-23:40; 2:30-3:10

ISC
(29-06-2025 01:00 - 29-06-2025 02:00) From remote
Status: Ended
Description: DIFFp loop oscillation in CARM_NULL_1F
Actions undertaken: Tuning the proper gain

Vacuum
(29-06-2025 06:00 - 29-06-2025 06:30) On site
Status: On-going
Description: check the level of nytrogen at WE

ITF found in LOW_NOISE_3 and in Science Mode. It unlocked at 13:32 UTC, and after a period of instability of the INJ system, it was possible to reach LOW_NOISE_3 again at 14:56 UTC.
The ITF unlocked again at 18:59 UTC, this time the IMC didn't relock. The INJ expert worked from remote to solve the issue and, from 20:50 UTC, the INJ system returned to its nominal state. 
Relock in progress.

Automation
INJ_MAIN crashed at 19:38 UTC during the recovery of the Injection system. It was possible to restart the process Via VPM at 19:45 UTC.

Guard Tour (UTC)
14:35 - 15:15
16:40 - 17:20 
19:56 - 20:42

Oncall events

ISYS
(28-06-2025 19:15 - 28-06-2025 20:50) From remote
Status: Ended
Description: At 18:59 UTC after the unlock the IMC cavity was not able to relock.
Actions undertaken: I've contacted the INJ Oncall, Melo, that from remote was able to restore the subsystem by manually positioning of the IMC mirror and then updating the setpoint of the BPC. System recovered at 20:50 UTC.

Two more glitches have been detected on WE, after one quiet day. The first, at 14 UTC (1435154659), looks very similar to the others: on one CAL channel, the third to last of the packet, we have 0.8. On LOCK FLAG we have the usual small glitch. The event occurred when the ITF was not locked; in case it was, the glitch was potentially destructive.

The second event, at 16_30 UTC (1435163616), occurred in observing mode and had no effect: all the visible channels show only a small distorsion, like a missing sample. Maybe the wrong data were present only in the last two channels, not visible and not important.

ITF found locked at LOW_NOISE_3 in SCIENCE mode.

At 07:01 UTC, ITF unlocked due to the unlock of IMC. I contacted ISYS expert to check th status of the system and informed Melo that the SIB1 LC was opened by the Guardian and Susp expert was informed on it.
07:10 UTC - TROUBLESHOOTING mode set.
I downloaded LC DSP and properly closed the control but unfortunately a DSP failure occurred (SIB1_Electr). SUSP expert fixed the problem.
09:15 UTC - ISYS again in standard state.

At 09:20 UTC I started to realign both cavities. At 11:00 UTC, with the help of ISC expert, we recovered the ITF standard state.
Relocked at LN3 at 12:39 UTC, after the usual cross-alignmnt in ACQUIRE_DRMI. AUTO_SCIENCE on.

Guard tour (UTC):
05:20 - 06:30
11:50 - 12:40

ISC
(28-06-2025 11:00 - 28-06-2025 11:45) Operator on site with expert from remote
Status: Ended
Description: Alignment recovery

ISYS
(28-06-2025 07:10 - 28-06-2025 09:18) From remote
Status: Ended
Description: IMC not relocking due to an opening of SIB1 opening loops

SUSP
(28-06-2025 07:10 - 28-06-2025 07:55) From remote
Status: Ended
Description: SIB1 LC opening and DSP communication problems. No correction signals.

ITF found in Science mode.

The ITF unlocked at 22:05 and the unlock seems to be an unlock of the IMC (see plot); after this event the  IMC went in a period of instability. In particular the IMC unlocked every time while performing fmoderr; the IMC in stand alone seemed to be more stable.

As a trial, at 22:40 UTC, I set fmoderr_skip = True in ITF_LOCK.ini and then the ITF was able to reach LN3 at the first attempt; still in Science at the end of the shift.

The parameter fmoderr_skip = False has been restored in the ITF_LOCK.ini file; to tested at the next relock.

Guard tours (time in UTC)

21:00-21:30; 23:10-23:50; 1:00-1:40; 3:00-3:30

ITF found in LOW_NOISE_3 and in Science Mode. It kept the lock for the whole shift.
ITF left locked.

Guard Tour (UTC)
17:48 - 18:28
20:51 - 

Since the operations of this morming  performed around 2025-06-27-09h29m12-UTC,  the Sc_WE debugging channels are  (see the plot)

• Sc_WE_noise  is equal to LSC_WE_LOCK_FLAG delayed
• Sc_WE_MIR_Z_CORR is equal to LSC_WE_CORR_SHAPED delayed
• Sc_WE_txAA  is equal to CAL_WE_MIR_Z_CORR delayed
• Sc_WE_tyAA is equal to LSC_B8_DC delayed
• Sc_Err_post is equal to  CAL_WE_MIR_Z_NOISE

The removal of two GLB channels have been performed this morning, both on NE and WE. The place of calibration lines in the list of the channels has been anticipated: now the last two are angular corrections, not used for standard operation ad temporarily disabled putting zero in the DSP gain. Those channels ar not visible in WE data; there are still two calibration channels (MAR has been rmoved). The probe remained free (Sc_WE_tyAA) is now used for the GLB channel corresponding to B8_DC.

So far, no glitch has been detected. Also LOCK FLAG, which uses to show a little distorsion when a glitch occurs, seems totally clean.

The glitches are due to events on WE marionette, involving the actuation on TX and TZ. A step is visible on both the correction; it induces a temporary misalignment, well visible on the optical levers. One can easily find a few of them going through the recent data.

There could be one vertical actuator which changes slightly its gain: the actuators are disposed in X configuration, which means that all are equally involved in the control of TX and TZ. The effect could be amplified by the fact that, after the mirror replacement, the correction needed to align TX remained a bit high (4 V). It can easily zeroed by means of the motors. It would be worth to do it and see if the glitches are mitigated.

In case they are not, it would be more difficult to exclude something more worrying, like a mechanical glitch, but I see it quite irrealistic.

ITF found in LOW_NOISE_3 and SCIENCE mode (Autoscience ON); BNS Range ~52 Mpc.
Unlocked at 06:57 UTC with excessive SRCL and MICH oscillations, high seismic activity on site.
Relock unsuccesful at LOCKING_CARM_MC_IR, REDUCING_CARM_OFFSET_3_OF_3 and three consecutive times at LOCKING_CARM_NULL_3F showing the same oscillations.
ITF parked in LOCKED_CARM_MC_IR at 07:43 UTC waiting for end of work on  QPD_B1p. Prealignment of PR-SR at 07:56 UTC with ITF set to TROUBLESHOOTING as suggested by Spinicelli, Was.
CARM_NULL_1F achieved at 09:01 UTC after decrease in DIFFp gain by Casanueva (on-call) see entry.
ITF set to DOWN at 09:27 UTC for DSP intervention by Bersanetti, Masserot, Ruggi and Was. Lock acquisition restarted at 09:36 UTC.
Back to CARM_NULL_1F on second attempt at 10:01 UTC after unlocking at LOCKING_ARMS_BEATING and again at CARM_OFFSET_STEP_2_OF_3.
LN3 achieved, back in SCIENCE at 10:39 UTC (Autoscience ON); same BNS Range.

DAQ
09:17 UTC Restart of SusDAQBridge by Bersanetti

DET
07:14 UTC Intervention on QPD_B1p Galvos and NE,WE MARionette calibration by Rolland.
DMS flags shelved under CalNorth and CalWest.

Figure 1. In the last few hours there was two glitches where the B1p power increased by a factor 2 for less than 1 second.

Figure 2 shows the trend over the last two days, and there is at least two more glitches like that. For example on June 25 just before midnight.