2026-04-02

The DAQ recovery, after the computing shutdown,  started on 2026-04-02-18h28m45-UTC once the CmNames server and the VPM server were restarted and that all the expected hosts (rtpc, olserver and stol) were alive .

We restarted  fisrt all the storage and data collection parts

• raw stream on stol01 and stol02  hosts
• rawfull, rds and trend streams on stol03 host 
• the data collection parts running  on
  • the olserver52 and olserver53 hosts
  • running on the rtpcs (TolmFrameBuilder)
• operations performed between 2026-04-02-18h28m48-UTC and  2026-04-02-18h38m44-UTC 

As everything seems to run correctly , we continued by restarting all the servers involved

• in the data access part 
• in the Detector Monitoring System 
• in the Automation part 
  • The PyHVAC server was unable to retart due to
    • the missing /dev/shm/zFbsIng imput SHM
    • and later the missing  input channels  V1:HVAC_INJ_TE_OUT_SET 
    • server fully recovered the  2026-04-07-08h27m37-UTC
• in the Image readout 
• operations performed  between 2026-04-02-18h39m15-UTC and 2026-04-02-19h08m44-UTC

We left all DAQ running during 1 hour to check that everything ran smoothly, thank to the availabilty of the DMS and VIM web pages.

 Then we continued by

• reconfiguring all the Tolm devices and by restarting all the rtpc's servers without closing the loops, except for those that are automatically closed by the ACL server.
• restarting the Telescreen 's servers
• restarting the environment's  servers in the Detector's Environnent Monitoring part 
• restarting the Newtow Noise servers
• operations performed between  2026-04-02-20h10m49-UTC and  2026-04-02-21h28m35-UTC

We left all the things running during the nigth to perform a more accurate check the next day

 2026-04-03

We found that a lot of Ethernet devices (lnfs, sqz, power suppplies ) not any more recheable .  

For  DAQ/DET ethernet devices

• the DaqBoxes  were reachable 
• the SBD power supplies were not reachable.
• the SDB internal switches were reachable 
• the SWEB QD  RD482-ETH bridge was not reachable, meaning the B8_QD{1,2}  shutters and Vbias were not anymore remotely driven and monitored

Thanks to the local support, the  SBD power supplies  remote monitoring/driving were recovered . 

To recover remote monitoring/driving B8_QD{1,2},

• the SWEB_PowerUnit12 was turned off/on and fortunately it was enough to recover this device.
• as the internal Timing mezzanine was switched OFF/ON to, the SWEB bench DBoxes were reconfigured 
• to recover the correct running conditions , the SWEB rach DBoxes were reconfigured too and the Acl's SWEB_rtpc servers restarted
• operations performed between 2026-04-03-09h19m33-UTC  and 2026-04-03-09h37m46-UTC

Easter break

During the Easter break, the olserver115 became unreachable around 2026-04-03-19h34m38-UTC . The MdVim server has been restarted temporaly on the olserver118  at 2026-04-04-04h20m30-UTC.

The olserver115 host has been recovered by the IT department and the MdVim server is now running on its host since  2026-04-07-06h32m25-UTC

2025-04-07

All the Fbs servers were restarted with a new release v8r24p0 

• operations performed between 2026-04-07-15h03m20-UTC and 2026-04-07-15h08m29-UTC

This morning we restarted the injection by:

- closing the EIB suspensions 

- restoring the previousl pump currents in the Neovan diodes (5 A and 4.9 A respectively)

- restoring the previous currents to the SL diodes (29.1 A each)

- locking PMC, closing BPC and manually aligning the IMC to reach the threshold for the lock and the automatic alignment to be engaged 

There is slighlty less power than before the shutdown, both in transmission of the Neovan and the PMC (4% of power missing).

From the camera the output shape and position of the neovan seemed ok (picture 1), so we went to the laser lab to align the PMC. With a very little tilt in the vertical direction we minimized the 01 peak, however the target power was not recovered (0.75 V vs 0.78 V).

The RFC has less power in transmission but we will wait for the ITF to be aligned because it has an influence on it.

This afternoon, a few hours after the restart of all CO2 lasers, thermal camera images of both CO2 benches were acquired in order to check the shape and position of the DAS rings/CH for both WI and NI.

In Figures 1 and 2, today’s images are compared with those acquired on 2026-03-10.

No changes in the shape or position of the actuators are observed compared to previous acquisitions.

CEB (BPC, IB, BS, NI, WI, PR, SR, SDB1) and MC SAT control systems have been recovered. BPC, IB and MC have been restored on the afternoon of last Friday to allow INJ subsystem recovery. The rest of CEB tower control systems have been restored today. Tomorrow we will proceed with the recovery of NE and WE SAT control systems.

This morning, an attempt was made to restart the LargeCoil processes in all three buildings via VPM, but it failed.
The issue was fixed by disconnecting and reconnecting the Ethernet cable of the coil, restoring proper communication.

Regarding the PR CHRoCC, I realized that I made an error: 2.796 V corresponds to the reference value for Vout, not to the VDAC.
The correct reference VDAC is 3.88 V.
Therefore, at 12:10 UTC, I set the VDAC to 3.88 V with a ramp time of 3200 s.

Here is the list of all actions performed to restart the TCS.

• RHs:

Restarting the process alone was not sufficient.
As usual, it was necessary to physically power cycle the power supplies. Therefore, Fabio went to the Central Building, then to the NE, and finally to the WE in order to perform this operation. After this intervention, it was possible to restart all the processes.  After that, all RHs were switched on again:

- SR RH: V=23.13 V at 08.54 UTC
- NE RH: V=16.6 V at 08.56 UTC
- WE RH: V=17.4 V at 08.59 UTC

• PR CHRoCC (the PR CHRoCC process was already restarted):

-The voltage was re-applied to the CHRoCC (V = 2.796 V, ramp time = 7200 s) at 08:35 UTC.

• CO2 LASERS:

The restart sequence was performed in the following order:

1. Main laser chillers
2. WI CH
3. NI CO2 main laser
4. NI CH
5. NI CO2 main laser

Nicola then re-enabled the Guardian system.
All operations were carried out between 09:00 and 09:30 UTC.

At around 10:08 UTC, Fabio returned to the TCS room because the DMS was showing the CO2 laser interlock-related channels as grey.
He disconnected and reconnected the network from the setup, after which the data were restored.

All TCS actuators have been switched on again.
The situation will be monitored over the next hours.

This morning we performed spot magnetic measurements in the area where the new environmental monitoring stations will be installed. Excavation works in this area are scheduled to start after the Easter break.

Measurements were carried out by placing the magnetic probe approximately at the center of the area and then repositioning it to different locations at a distance of about 8 m from the center within the marked perimeter, Figure 1. The magnetic field was measured using a Bartington Mag-03MC100 sensor (range ±100 μT, calibration factor 100 mV/μT), connected to a portable data logger. The probe was consistently oriented with its z-axis (ch-3) pointing towards North during the campaign. For each position, spectral measurements were acquired and stored.

During the measurements performed at WEB and CEB, we observed the presence of a noise source, switching on and off in a non-regular manner.

This behaviour was identified through the magnetic spectra: when the source was not active, only the X-axis (ch1-x) showed a significant reduction of the noise floor, reaching values of a few pT/√Hz for frequencies above ~90–200 Hz. The Y and Z axes did not show any noticeable variation and remained unchanged regardless of the source activity, Figure 2.

The fact that this feature was not observed in the North position could be related to the source being continuously active during that specific measurement. 

It is worth mentioning that agricultural activities were ongoing in the vicinity of the site during all the measurements.

In addition, distinct spectral features are observed around ~23 Hz and ~32 Hz, consistently present in all the acquired measurements, Figure 3.

Below, we report the spectral amplitudes of the main power-related lines at 50 Hz and its harmonics (150 Hz and 250 Hz). The values have been converted into magnetic field units using the sensor calibration factor, corresponding to 1 μV = 0.01 nT. 
For some positions, measurements were taken both when the magnetic noise source was active and when it was inactive, since its disappearance was noted during data acquisition, thus allowing both conditions to be recorded. In addition, dedicated measurements were performed by placing the probe in proximity to the external magnetometers.

***************** NEB ************************************

********************* WEB**************************

************************ CEB **************************************

***************External magnetometers*****************

In line with the planning detailed here:

https://indico.igwn.org/event/7/

Today we proceeded with the 2nd Computing core services shutdown starting at 12:00 and with UPS electrical shutdown in the Control Building lasting one hour starting around 13:00.

The new UPS serving the Computing Room has been put in operation and Computing core services has been progressively restored during the afternoon and made available around 18:00.

To proceed quickly with some test of the authentication chain the Cm and VPM hosting machines (olserver111 and olserver113) has heen restarted and the Cm servers for Cascina, CascinaLV, CascinaTest, CascinaLVTest (and their corresponding VPM servers) have been restarted without problems.

The rest of detector computing and storage services are being restarted with the goal to be ready early tomorrow morning to resume the DAQ chain possibly until making operative the Detector Monitoring System.

On Monday, March 30, we transported the WI PA actuator from Rome Tor Vergata to the site. Thanks to Fabrizio, we unloaded the actuator at the entrance of the Central Building, leaving it there until the following day before moving it to the TCS room.
During the maintenance day on Tuesday, March 31, we mounted the lifting frame used to sling the actuator to the overhead crane (Fig. 1). Fabio then helped us operate the crane and move the actuator toward the TCS room (see Fig. 2).
We placed the actuator on a trolley and moved it inside the TCS room. There, we installed the heater source and its paraboloid inside the actuator structure (Fig. 3), then left it in the TCS room properly covered with plastic film (Fig. 4).

The ITF was found in DOWN and UPGRADING mode with the controlled shutdown in progress.

All subsystems were placed in a safe state by their respective experts, and the DAQ was shut down at approximately 16:00 UTC.

All the DAQ servers have been stopped . Operation complete at 15h41m-UTC 

Note:  the DaqBox server running configurations stored in ~virgorun/.local/share/DaqBox have been moved in the ~virgorun/.local/share/DaqBox/20260325-ComputingShutdown directory to ensure the full reconfiguration of these DaqBoxes at the recovery time 

We completed the Acl v1r39p17 version deployment 

• SQZ rtpcs: rtpc2 (trend plot), rtpc3 (trend plot) and rtpc23  (trend plot)
  • All the LC and SBE loops were already openned by Marco between the  2026-03-23-23h58m42-UTC and 2026-03-23-23h59m27-UTC
  • Operations performed between 2026-03-24-07h11m30-UTC and 2026-03-24-07h22m09-UTC
• INJ rtpc: rtpc19 (trend plot)
  • The EIB bench has been blocked by Matthieu around 7h10-UTC
  • Operations performed between 2026-03-24-07h22m46-UTC and 2026-03-24-07h26m43-UTC

The Acl update on the SUSP rtpc(rtpc4) has been postponed after the computing shutdown, in agreement with the TCS and commissioning coordinators

ITF Found in LOW_NOISE_3_ALIGNED State and COMMISSIONING Mode.
All times are UTC.
07:00 ITF manually in DOWN state to start the powering off of the ITF for the upcoming controlled shutdown.
07:13 Earthquake in Fosdinovo (MS), ML 4.0 (INGV);
08:27 Set PAUSE to nodes: ITF_LOCK (Operator), SQZ_FLT (Operator), and DET_MAIN  (Gouaty);
08:32 Opened position loops via VPM of: SIB2_SBE, SDB2_SBE, SNEB_SBE, SWEB_SBE, since they opened themself between 7:44 and 7:50 maybe related to the Earthquake in Fosdinovo (MS);
08:44 UPGRADING Mode set;
08:52 INJ OFF (INJ PMC FLIP set to "UP") (Gosselin);
8:55 - 09:03 TCS Lasers all off (Melo) and Chillers Guardian system off (Ballardin), see entry #68947; 
09:10 - 10:38 Shelved TCS Flags in DMS: mean_TCS_NI_CO2_PWRLAS_10, mean_TCS_WI_CO2_PWRLAS_10, mean_TCS__GUA_WI_FLOW_10, mean_TCS__GUA_NI_FLOW_10, mean_TCS__GUA_BCK_FLOW_10, TCS__GUA_Rules, mean_TCS__GUA_BCK_TE_10, mean_TCS__GUA_NI_TE_10, and and mean_TCS__GUA_WI_TE_10 up to 07-04-2026 09:00 UTC (Operator under request of Nardecchia);
09:46 NI/WI Etalon loops open (Operator);
09:54 Opened SBE_EIB position loops: PID and Actuation (Operator);
09:58 Set PAUSE to nodes: OMC_LOCK, and ITF_CONDITIONS (Operator);
10:29 - 10:37 Automation Metatron nodes VPM process turned off (Bersanetti);
The larger part of the VPM processes has been shutted down during the morning and the afternoon;
ITF left in DOWN State and UPGRADING Mode;

Relevant logbook entries reporting switchoff sequences:
- TCS actuators in CB switched OFF (#68947, and #68947);

WE RH  and NE RH have been turned off at 14.28 UTC and 14.29 UTC, respectively.

Immediately after it the related processes on the VPM have been stopped as well. 

Switch-off sequence:

• 08:22 UTC – SR RH OFF
• 08:24 UTC – PR CHRoCC OFF (7200 s ramp)
• 09:00–09:02 UTC – CO₂ CH lasers OFF
• 09:02–09:03 UTC – CO₂ main laser OFF
• 09:03 UTC – Guardian system disabled and CO₂ chiller OFF

Note: outputs disabled; power supplies left ON.

We checked that the loops were working well after the ACL update.

We sent SQZ_MAIN SQZ_LOCKED_NO_FC and it went at the first attempr

We then aligned the filter cavity but we had to modify some of the code lines in the FC Alignment interface.

We locked the FC both with only the mirror and also with the laser. We updated the gains. SQZ_CC GAIN=4 and LFC_Z_GAIN = 6 instead SQZ_CC GAIN=1.55 and LFC_Z_GAIN = 6

We also run a SQZ CC Phase can with SR tuned at 17:34 UTC CC GAIN 40000. We also noticed that the CC sensing noise is dominant above 100 Hz

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Once the detuning of SR started we open the shutter, we set 4.1 rad of SQZ angle and we run autoalignment loop for more than one hour. Then we did the following measurement:

• 20:24 UTC phase scan, some issue in the phase
• 21:07 UTC a\other phase scan and a long stretch of shot noise
• 22:16 UTC SQZ injection with phase equal to 4.25 rad
• 23:44 UTC stop injection and AA loop dither line

ITF found in COMMISSIONING, LN3_ALIGNED
17:00 UTC tuned squeezing phase scan (Vardaro)
18:05 UTC detuned phase scan: SRCL set set to 11
18:30 UTC HVAC oncall called due to a boiler failure at 1500W. Intervention scheduled for tomorrow morning, flag shelved for the night.
22:00 UTC At the end of the shift I left the ITF in COMMISSIONING, LN3_ALIGNED

The following report has been submitted to the On-call interface.

On-call events -> Air Conditioning

Title: WAB boiler failure

Author(s): andreazzoli

Details:

Due to the oil-fired boiler shutting down, the water temperature in the hot water circuit has dropped to 20 °C (set to 50 °C).
Boiler operation has been restored. The system is operational but requires a more thorough inspection, which will be carried out tomorrow morning. Soldani D. has been informed.

* Note that any files attached to this report are available in the On-call interface.

ITF Found in LOCKING_ARMS_BEAT_DRMI_1F State and COMMISSIONING Mode.
All times are UTC.
07:00 MAINTENANCE started.
07:09 Acl Update step 3 Started (Masserot).
07: 20 INJ beam blocked for INJ Acl Update (Gosselin, remote), then I requested DOWN to INJ_MAIN Metatron node and I paused ITF_LOCK.
07:27 Acl Update step 3 Completed for INJ and SQZ, while TCS, and SAT postponed afted the shutdown (Masserot, #68939).
07:35  INJ beam block removed (Gosselin, remote).
07:43 INJ in IMC_RESTORED.
10:43 ITF in LOCKED_ARMS_IR.
11:00 MAINTENANCE ended.
12:57 ITF in LOW_NOISE_3_ALIGNED;

Task performed by the operator during the maintenance:
- From 08:25 to 8:35 OMC Lock;
- From 08:38 to 08:59 OMC Scan;
- 09:08 TCS Chillers Refill;
- 09:18 TCS Thermal Camera Reference;
- 11:15 TCS Power Checks at the Pickoff:

This line's amplitude continuously decreased and cannot be clearly seen after midnight today.