Virgo Logbook

Virgo Runs (O4c)

menzione - 6:51 Monday 14 July 2025 (67282)

Operator Report - Night shift

ITF found locked in LN3 and in SCIENCE mode.
At 23:33 UTC, no more data for all experimental devices at WE are available, including SUSP, minitower, racks, ENV sensors, RH, Coil driver, MotSwitch, FlipMirror, PCal, ALS and all VAC channels.
I assumed a problem on WE UPS (also not available), so I contacted Electricity expert,
D'Andrea went in WE building so I set TROBLESHOOTING mode set from 00:19 to 01:00 UTC. But after a check, he verified that all Electricity systems were ok. So I contacted OnLine&Computing OnCall. After an investigation from remote, Cortese decided to came on site to check the status of the WE fiber switch. The switch was ok but few fibers were gnawed by mice.
At 01:28 UTC ITF unlocked. WE susp and SWEB loops opened. Not closables.
All servers and all processes for WE was no more reachable. W cavity misaligned. Metatron nodes in Cm error. GAME OVER.
01:33 UTC - TROBLESHOOTING mode set.

After the replacing of fibers, carried out by Cortese, I started to recover the DAQ server lost and, with the help of Masserot, we are recovering trying DAQBoxes. Unfortunately we have still networking communications problems.
Computing and DAQ task forces at work...

Guard tour (UTC):
23:32 - 00:11
01:00 - 01:40

Oncall events

DAQ
(14-07-2025 03:15 - ) Operator on site with expert from remote
Status: On-going
Description: I restarted several servers and recovered several processes but not all. In particular the daq boxes.

Electricity
(14-07-2025 00:00 - 14-07-2025 01:00) On site
Status: Ended
Description: WE UPS check. UPS ok.

On-line Computing & Storage
(14-07-2025 01:00 - 14-07-2025 03:00) On site
Status: Ended
Description: WE fibers gnawed by mice.
Fibers replaced successfully.

Images attached to this report

67282_1752467707_Screenshot from 2025-07-14 06-34-50.png

67282_1752467711_Screenshot from 2025-07-14 06-34-14.png

Comments to this report:

masserot - 8:58 Monday 14 July 2025 (67286)

The optical fibers between the CEB-DAQroom and the WEB used for the Tolm network do not work, those using the Tolm v1 format for suspensions, as well as those using the Tolm v2 format used by the DAQ boxes

Below the detail of the connections between the MxDx and the patch panel at WE

Rack_E4-TDB_v1_SN09A - multimode fiber - PP-CEB-03 (10MHz)
Rack_E4-TDB_v1_SN09A - multimode fiber - PP-CEB-04 (IRIGB)
Rack_E4-MxDx_v2_SN10-LK1 - monomode fiber - Rack_E4-PP-WEB-RACK-WALL - PP-11_12 - PP-WEB-CEB-11_12
Rack_E4-MxDx_v2_SN10-LK3 - monomode fiber - Rack_E4-PP-WEB-RACK-WALL - PP-09_10 - PP-WEB-CEB-09_10
Rack_E2-7slot-VME-MxDx_v1_SN12-LK6 - monomode fiber - PP_WEB-CEB-21-22
Rack_E2-7slot-VME-MxDx_v1_SN12-LK7 - monomode fiber - PP_WEB-CEB-23-24

cortese - 10:43 Monday 14 July 2025 (67290)

It turned out that other patch panels at WEB were visited by mice affecting the DAQ connections.

At about 10:15 LT I connected also the DAQ fibers from CEB to WEB with spare fibers moving them in this way:

Multimode PP-WEB to CEB: from 3-4 to 5-6 (note that in this case the PP doesn't show fibers visibily eaten)
Monomode PP-WEB to CEB:
- from 9-10 to 7-8
- from 21-22 to 15-16
- from 23-24 to 17-18

Since there are no more momomode spare fibers left in the WEB to CEB patch panel the last fiber couple has been rerouted in this way:

Monomode PP-WEB to CEB: from 11-12 to PP-WEB to COB 13-14 then to PP-COB-to-CEB3 13-14

cortese - 11:50 Monday 14 July 2025 (67291)

After checking with Alain it turns out that among the monomode fibers only the Lk3 link is working (couple 7-8) while multimode links are working.

Since there are no spare monomode backbone fibers anymore at WEB we are scheduling an urgent intervention by an external firm to reconnect and check at least 24 fibers.

cortese - 11:58 Monday 14 July 2025 (67292)

Monomode fiber reconnection intervention at West End Building is scheduled for tomorrow Tuesday at 8:30 LT