ITF found with ITF_LOCK, ARMS_LOCK in DOWN and paused, DRMI_LOCK in MISALIGNED_PR_SR and paused with TROUBLESHOOTING mode, due to WE fiber connection loss. Thus no activity planned for this afternoon (calibration with ITF unlocked and ISC injections) can be carried out and night shift will be cancelled.
19:25UTC closed 1500W building entrance 2 (fig.1).

Guard tour (UTC)
17:53 -> 18:28
21:15 ->

ISYS
13:40 - 14:50UTC Measurement of PSTAB TF in LaserLab by Spinicelli to investigate recent INJ unlocks. IMC unlocked at 14:18UTC, recovered at 14:45UTC.
16:28 - 16:47UTC check on IPATSiA by Tringali and Spinicelli. Nothing anomalous to report except 1500W building entrance n.2 (towards WE) found wide open.
IMC unlocked again at 16:44UTC, opened PSTAB loop at 16:56UTC under request by Spinicelli.

In order to debug the recent problem of stability of the PSL/INJ system, we measured the TF of the PSTAB (fig. 1) and the error signal of the PMC loop (fig. 2).

At a first sight, we couldn't spot any strange behaviour of those part, but we will keep investiganting tomorrow morning.

We have tuned the amplitude sent to the DAC channel for the NEB automated acoustic injections. 
The value of the amplitude parameter in the ini.file is Amp= 0.1 (purple curve) 
The same should be applied to WEB automated acoustic injections.

We have left the loudspeaker connected to the PDU and ready for automated injections.

VPM process: AcousticInjectionNEB

Configuration parameter: /virgoData/NoiseInjections/MagneticInjectionsO4/EnvSwepts/config/ConfigurationAcousticColoredNEB.ini

The following report has been submitted to the On-call interface.

On-call events -> Electricity

Title: Control network UPS building 3000 West

Author(s): dandrea

Details:

I was called by the operator on duty who informed me that he could no longer connect with the devices in building 3000 West.
I have been asked to intervene for a suspected anomaly on the UPS electrical network.
I intervened on the site and could verify that the UPS of the terminal building were working regularly, then I inspected the interior of the building to check for any interruptions on the secondary distribution but everything was regularly fed.
I deduced that it was a problem related to the data network, so I did an inspection at the network switches of the terminal building and also at the intermediate building West, but everything was in regular operation.
I told the operator that the problem was with the data transmission.

* Note that any files attached to this report are available in the On-call interface.

Monomode fiber reconnection intervention at West End Building is scheduled for tomorrow Tuesday  at 8:30 LT

After checking with Alain it turns out that among the monomode fibers only the Lk3 link is working (couple 7-8) while multimode  links are working.

Since there are no spare monomode backbone fibers anymore at WEB we are scheduling an urgent intervention by an external firm to reconnect and check at least 24 fibers.

It turned out that other patch panels at WEB were visited by mice affecting the DAQ connections.

At about 10:15 LT I connected also the DAQ fibers from CEB to WEB with spare fibers moving them in this way:

• Multimode PP-WEB to CEB: from 3-4 to 5-6 (note that in this case the PP doesn't show fibers visibily eaten)
• Monomode PP-WEB to CEB:
  • from 9-10 to 7-8
  • from 21-22 to 15-16
  • from 23-24 to 17-18

Since there are no more momomode spare fibers left in the WEB to CEB patch panel the last fiber couple has been rerouted in this way:

Monomode PP-WEB to CEB: from 11-12 to  PP-WEB to COB 13-14 then to PP-COB-to-CEB3 13-14

Here I attach the two plots that I refer to in the entry.

The following report has been submitted to the On-call interface.

On-call events -> Interferometer Sensing & Control

Title: CH tuning

Author(s): casanueva, cifaldi

Details:

Since the situation of the interferometer was so unstable, we decided together with Michal to make a step of the CH already on Sunday. Based on past experience we made a 10% common decrease of the CH. We waited for 1 hour and we tried to relock. The lock acquisition was succesful at the second attempt. FIgure 1 shows the evolution of the DIFFp TY gain, which is significantly better than in the morning.

Also FIgure 2 shows the evolution of the sidebands in the CITF during the CH transient. There is not a significant variation, which is encouraging.

* Note that any files attached to this report are available in the On-call interface.

The following report has been submitted to the On-call interface.

On-call events -> Air Conditioning

Title: WEB LPG Boiler down

Author(s): soldani

Details:

Alarm received from DMS; boiler blocked, reporting an electronic control mainboard error;
The alarm was cleared and reset, and the boiler restarted correctly. A technical intervention for Maintenance Service is required for Tuesday, July 15, to check and possibly replace the electronic board.

* Note that any files attached to this report are available in the On-call interface.

The optical fibers between the CEB-DAQroom and the WEB used for the Tolm network do not work, those using the Tolm v1 format for suspensions, as well as those using the Tolm v2 format used by the DAQ boxes

Below the detail of the  connections between the MxDx and the patch panel at WE

• Rack_E4-TDB_v1_SN09A  - multimode fiber - PP-CEB-03 (10MHz)
• Rack_E4-TDB_v1_SN09A  - multimode fiber - PP-CEB-04 (IRIGB)
•  
• Rack_E4-MxDx_v2_SN10-LK1   - monomode fiber - Rack_E4-PP-WEB-RACK-WALL - PP-11_12 - PP-WEB-CEB-11_12
• Rack_E4-MxDx_v2_SN10-LK3   - monomode fiber - Rack_E4-PP-WEB-RACK-WALL - PP-09_10 - PP-WEB-CEB-09_10
•  
• Rack_E2-7slot-VME-MxDx_v1_SN12-LK6 - monomode fiber -  PP_WEB-CEB-21-22
• Rack_E2-7slot-VME-MxDx_v1_SN12-LK7 - monomode fiber -  PP_WEB-CEB-23-24

ITF found locked in LN3 and in SCIENCE mode.
At 23:33 UTC, no more data for all experimental devices at WE are available, including SUSP, minitower, racks, ENV sensors, RH, Coil driver, MotSwitch, FlipMirror, PCal, ALS and all VAC channels.
I assumed a problem on WE UPS (also not available), so I contacted Electricity expert,
D'Andrea went in WE building so I set TROBLESHOOTING mode set from 00:19 to 01:00 UTC. But after a check, he verified that all Electricity systems were ok. So I contacted OnLine&Computing OnCall. After an investigation from remote, Cortese decided to came on site to check the status of the WE fiber switch. The switch was ok but few fibers were gnawed by mice.
At 01:28 UTC ITF unlocked. WE susp and SWEB loops opened. Not closables.
All servers and all processes for WE was no more reachable. W cavity misaligned. Metatron nodes in Cm error. GAME OVER.
01:33 UTC - TROBLESHOOTING mode set.

After the replacing of fibers, carried out by Cortese, I started to recover the DAQ server lost and, with the help of Masserot, we are recovering trying DAQBoxes. Unfortunately we have still networking communications problems.
Computing and DAQ task forces at work...

Guard tour (UTC):
23:32 - 00:11
01:00 - 01:40

DAQ
(14-07-2025 03:15 - ) Operator on site with expert from remote
Status: On-going
Description: I restarted several servers and recovered several processes but not all. In particular the daq boxes.

Electricity
(14-07-2025 00:00 - 14-07-2025 01:00) On site
Status: Ended
Description: WE UPS check. UPS ok.

On-line Computing & Storage
(14-07-2025 01:00 - 14-07-2025 03:00) On site
Status: Ended
Description: WE fibers gnawed by mice.
Fibers replaced successfully.

The following report has been submitted to the On-call interface.

On-call events -> Network

Title: Network backbone link interrupted between COB and WEB buildings

Author(s): cortese

Details:

Called by Antonella Bozzi after the escalation of the problem was suggesting a networking issue.

The interruption was caused by the failure at 1.30 AM LT of all the 4 monomode fibers connecting network star center switch in COB to West End Building switch.

The failure is due to a mouse eating fibers 1,2,3,4,23,24 directly in the network rack at the north-east corner of the WEB building.

The network backbone connection has been recovered around 4:45 AM LT using spare fibers.

Also the Hartmann Sensor monomode fiber connection was broken and moved to another spare couple

However only one of the two redundant network backbone links turns to be working. This means that one of the spare fibers does not work too.
The redundancy will be recovered on the next tuesday maintenance.

* Note that any files attached to this report are available in the On-call interface.

The following report has been submitted to the On-call interface.

On-call events -> Network

Title: Network backbone link interrupted between COB and WEB buildings

Author(s): cortese

Details:

The interruption was caused by the failure at 1.30 AM LT of all the 4 monomode fibers connecting network star center switch in COB to West End Building switch.

The failure is due to a mouse eating fibers 1,2,3,4,23,24 directly in the network rack at the north-east corner of the WEB building.

The network backbone connection has been recovered around 4:45 AM LT using spare fibers.

Also the Hartmann Sensor monomode fiber connection was broken and moved to another spare couple

However only one of the two redundant network backbone links turns to be working. This means that one of the spare fibers does not work too.
The redundancy will be recovered on the next tuesday maintenance.

* Note that any files attached to this report are available in the On-call interface.

ITF remained locked at LOW_NOISE_3 in SCIENCE mode (Autoscience ON) for the whole shift. BNS Range ~53 Mpc.

Guard tour (UTC)
          -> 13:09
14:27 -> 15:07
16:30 -> 17:11
20:05 -> 20:45

Figure 1 shows the SR TX SET tests in the past 10 days. The main point of this figure it is too higlight when these tests happened, but also that the automation needs to be updated to have a reset of SR TX SET in DOWN in DRMI_LOCK with an explicit ramp time. Othewise the ramp down to zero takes 2 hours, and maybe perturbing the following lock acquisition.

Figure 2 shows the correlation between the BNS range and SDB2 B1p QD2 V. The quadrant signal is affected by the SR TX misalignment, as that changes the pattern on the camera, and seem to have maximum around -0.05V, which is different from the typical value around -0.15 when there is no SR TX SET 

The removed air filters have been cleaned up and re-installed on the morning of Friday 11th.

In order to investigate the glitches on Sa_BS, causing unlocks, on Thursday 10th evening we rebooted the crate. After that intervention the number of glitches on the signals generated by coil driver board connected to F0 (serial #41033) increased dramatically. To temporarily reduce the unlocks we enabled the keep sample feature on both the master and coil driver boards. We therefore decided to replace the board on the morning of Friday 11th with a spare (serial #45267). After the intervention the glitches disappeared even though the keep sample is currently disabled. We profited from Friday intervention to reset the master board of Sa_PR that has an anomalous delay in trasmission to DAQ.

ITF found in Troubleshooting.
After a couple of attempts and the intervention of M.Pinto to avoid an unlock caused by BStx PRtx oscillation, ITF reached LN3 at 06:27UTC
Science mode set at 06:29UTC

At 07:54UTC ITF unlocked again. As observed yesterday, several unlocks occurred at different steps during the lock acquisition
After reviewing the data related to last night's unlocks, Julia noticed that the Diffp at CARM NULL 1F was different. A similar issue had previously been resolved by reducing the power levels on both CHs.
In agreement with Michal, at 10:45UTC I contacted M. Cifaldi to reduce the power by 10%.

After waiting for the thermal transient, Julia restarted to work on locking acquisiton at 11:41UTC.
Finally, ITF reached LN3 at 12:41UTC.

DET
09:03UTC, SDB2_B1p_QD2 Vbias rearmed during the lock acquisition.
09:58UTC, SDB2_B1p_QD1 and SDB2_B1p_QD2 Vbias rearmed during the lock acquisition.
10:15UTC, SDB2_B1p_QD1, B1p_QD2 and B5_QD2 Vbias rearmed during the lock acquisition.
10:57UTC, SIB2_B2_QD2 galvo loop properly closed after opening.
12:01UTC, SDB2_B1p_QD1 and SDB2_B1p_QD2 Vbias rearmed during the lock acquisition.

Guard Tours (UTC):
05:59 - 06:38
07:57 - 08:37
10:28 - 11:06
12:28 - 

Here the plots that I mention in the entry.

The following report has been submitted to the On-call interface.

On-call events -> Interferometer Sensing & Control

Title: Unstable ITF

Author(s): casanueva

Details:

This morning the interferometer locked once but once it unlocked it doesn't manage to lock again. Same pattern as yesterday. Looking more closely the difference between today's lock acquisition and three days ago, the only thing that strikes me is that the DIFFp servo behaves completely different during the timer of CARM NULL 1F (Figures 1 and 2).

Something simialr happened already a couple of weeks ago (#67170), and I remember that a long time ago we used this behaviour of the DIFFp servo to fine tune the CH powers, to minimize the thermal transient. I am wondering if the external temperature drop can have pushed the ITF into a slighlty different state so that the CH needs to be readjusted to minimize the transient. To be better understood.

* Note that any files attached to this report are available in the On-call interface.

The following report has been submitted to the On-call interface.

On-call events -> Interferometer Sensing & Control

Title: Systematic unlocks of the ITF

Author(s): casanueva

Details:

Nicola contacted me because the ITF was unlocking systematically while passing to DRMI 3F. I tried to diagnose the problem, and I slightly tuned the phase of the signal. However the interferometer kept unlocking further in the lock acquisition. I tried to propagate the demodulation phase through other steps but still the interferometer is very unstable and keeps unlocking at different steps. Even during the timer of CARM NULL 1f, it unlocks, while there it should have already reached a sort of steady state. The main culprit seems MICH that oscillates at 11Hz, which is normally sign of a coupling between the MICH and SRCL loop when the first has too low gain and the second has too high gain. However the servo loops seem to work properly.

After TCS checked the powers to be correct we run out of ideas and we let the ITF for the night.

* Note that any files attached to this report are available in the On-call interface.

I was called by Ettore to check some data after the opening of the WI top stage controls of tonight (loops properly closed by him). While I was checking the data I started to look also at the failed locking attempts of tonight. There was one attempt which died at ACQUIRE LN2 due to the BStx PRtx 1.2 Hz oscillations.

At the next lock, while the BS was closing in FBW the oscillation arised again so (since the ITF was in troubleshooting) I manually put in drift control before a too high divergence of the oscillation. After a while, reaching LN3, I restored the FBW control. Now we are in nominal configuration. I didin't modify anything in the automation.

Today, I found ITF in TROUBLESHOOTING, we have several different problems with the locking acquisition. J. Casanueva was contacted by N. Menzione to solved the issue, she had been working on it for some time, so together, we decided to contact the Commissioning Coordinator, M. Was, to determine the next steps. To exclude other potential issues, I also reached out to M. Cifaldi to check whether the TCS could be involved. After a review, M. Cifaldi confirmed that the TCS was working as expected. After numerous attempts, during which we varied certain parameter values, we decided (together with Michal and Julia) to gather additional unlock data for further analysis. For some reason ITF almost reached LOW_NOISE_1 but it got stuck in a loop trying to tuned the Fmoderr it remained for 40 minutes like that and I called C. Derossi for help, but it was an unusual behavior so we dediced to put it DOWN and restarted the lock. I reverted the change on the ITF_LOCK ini file from fmoderr_skip = true back to fmoderr_skip = false. As agreeded with J. Casanueva and M. Was, after gathering others unlock data, I put ITF in LOCKED_ARMS_IR, to keep the alignment. Issue still present.

ISC
(12-07-2025 19:20 - 13-07-2025 01:30) From remote
Status: Ended
Description: CITF unlocks systematically