Venting of SR tower is in progress

At 20:05 UTC, the SR ring heater was switched off.

ITF found locked on ARMS_IR in UPGRADING mode and TCS crew woring "Power and centering checks".
Activity concluded at 14:30 UTC. ITF recovered up to LN3_ALIGNED.

At 19:52 UTC we decided to unlock.
Action performed:

* put several metatron nodes in PAUSE (DET_MAIN, ITF_LOCK, ARMS_LOCK, DRMI_LOCK, NARM, WARM, ALS_NARM, ALS_WARM and SUS_SR).
* SDB2_LC: Open Angular Loop
* SDB2_SBE: Open Position Loop
* SDB1_LC : Open Angular Loop 
* SR loops opened (ID, F7 and LC) for the SR venting planned for tomorrow.
* SR RH switched OFF by Corubolo.

Today we continued the work on the recovery of the lock acquisition, resuming from yesterday's state.

Initially during the morning we could see some issues related to the engagement of the SSFS, both at the immediate engagement of the loop and after the loop was closed; to fix the former a restart of the SSFS_Ctrl process was sufficient (although it happened again during the afternoon), while for the latter there is the known link with the IMC working point, so the overall IMC working point was checked (see entry 68978).

We still could not manage to stay in CARM_NULL_3F long enough to understand what was the main issue.

At the time of the daily meeting we discussed about possible next steps. One was to check (again, as it was promptly done at the restart of the interferometer a few days ago) the alignment of the TCS actuators, as some recent changes in chillers/temperature behaviour made us think about this correlation. The actuators were checked (see entry 68982) and everything was as expected.

Another issue could be due to the glitches (and anomalous behaviour) seen on the MC suspension controls that was observed yesterday. A check about that will be done on Monday, when no interferometer will be available anymore.

We then refocused on the lock acquisition, and we observed that the angular control of the input beam was still needed to be recovered. Fixing that, and managing also to fix a couple of the handoffs from 3F to 1F, we started to work consistently in CARM_NULL_1F, commenting out several pieces of the acquisition and restoring them one by one. We could then check and reasonably fix the phases for all of them: DARM, PRCL, MICH/SRCL, SSFS, DIFFp, PR and IB. One fully automated acquisition was done with no problem.

Then we moved to the DC readout acquisition, despite the dark fringe being a little brighter then usual (but not in a worrying way); the lock of the OMC, and everything else after that, went smoothly, and we could recover LOW_NOISE_3_ALIGNED at the first attempt at 18:50:32 UTC.

The performance is not great (~ 36-39 Mpc), but there are several things missing: the etalon loops are open, the squeezing is not recovered and its shutters open towards the ITF, and we haven't thermalized yet. Other small things related to performance (subtractions, actuators balancing and so on) were not checked at all, as it wasn't the point for this brief recovery.

After ~ 1 hour of data, we unlocked and we proceeded to prepare for the next few weeks of hardware interventions and no interferometer, starting with the venting of the SR tower:

• all automation DOWN; all ISC nodes, DET_MAIN and SUS_SR paused; all other SUS nodes, OMC_LOCK and INJ_MAIN in standalone mode;
• controls of SDB2 SBE/LC open, SDB1 LC open, SR suspension open;
• SR ring heater turned off.

Since the ITF unlocks at the very beginning of CARM_NULL_1F (68979), the powers and the centering of the CO2 actuators were checked to assess any variations.

Between 13:35 and 14:00 UTC, different images were acquired using the thermal camera. No displacement was detected, based on the actuators’ conditions on 10 March 2026 (see Figure 1).

From 14:00 to 14:15 UTC, the power of the CO2 actuators was verified. All measured values are consistent with the nominal values, as reported in the following table.

We estimated the hrec noise that the magnetic noise generated in the hall during the grounding injection would produce according with the coupling (CF) of ambient magnetic fields measured with big coils injections. The estimate is based on CEB magnetometers (Metronix).

Figure 1 shows the magnetic field modulus measured during the comb grounding injection (20 Mar 11:18 LT, 0.5 A). Figure 2  shows the measured Coupling Function using a fit of all O4 measurements, the colors correspond to the 95% confidence interval (the region that contains 95% of the measured points). Figure 3 compares the measured (blue) and projected noise (other colors). As guessed, predicted values are at least a factor 100 below measured ones.

Some considerations:

• the distribution of the generated magnetic field across the hall is quite non-uniform, as shown in the magnetometers picture of elog 68927, magnetic field at the BS, NI and DT area is larger than at INJ tower respectively by a factor factor 100, 40 and 10. Instead, when the big coil is used for injections, the field distribution is uniform within a factor of 3 over the entire CEB hall: see Figure 4 depicting the CEB hall magnetometers response during the sweep injection of August 18 2025.  
• A reason for such non uniformity could be the one indicated by Federico in his comment : elog 68886 (our grounding injection circuit forms a sizeable loop located in the area NI-SR).
• For this reason, we think that the CF measured with O4 sweeps cannot be assumed a good representation of the hrec impact of the magnetic noise produced by our grounding injection.
• We cannot exclude magnetic coupling based on this projection.
• From a rough computation, if we were applying instead CF measured with the Bartington magnetometers located close to BS and NI towers (where, as from Federico's comment, the "injection circuit loop" is located, the project noise would be more or less a factor 10 larger than that shon in Figure 3. 
• A last observation concerns the slope of the magnetic noise projection shown in Figure 3: in the 10-100 Hz range the measured and predicted slopes are the same: approx 1/f^3.
• It might be interesting to evaluate predictions from magnetic CF measured with close field injections (small coil on towers base), for example nearby NI, BS and DT.
• We could learn from repeating grounding injections from different locations, also reducing the circuit size (shorter wires), with the intent of reducing the injected magnetic fields.

ITF  found in Upgrading mode and in locked arms.

The shift was dedicated to the ITF recovery; at the moment the ITF unlocks at the bvery beginning of CARM_NULL_1F without any clear reason.

To understand this behaviour the commissioning team performed the following parallel checks:

• checks on the IMC loop
• TCS power and thermo camera images chesks (TCS expert from remote)

Investigations in progress.

Software

BacnetServer restarted at 12:03 UTC because it was providing flat data.

This morning, since the ITF unlocks once the SSFS is engaged, we checked the longitudinal and angular working points of the IMC loop.

• For the longitudinal we found that the offset which minimize the coupling of the fmoderr line on the pmc is -0.07 V (instead of 0.03 V, so no much difference), see attached plot.
• For the angular wp  (offsets on the galvos used to center the beam on the RF quadrants), we found that the did not change substantially:

ACL_CONST_CH FF_H_offset    "V/V"    1   LOOP_FREQ   -0.25 #-0.28
ACL_CONST_CH FF_V_offset    "V/V"    1    LOOP_FREQ   +0.5 #+0.58  
ACL_CONST_CH NF_H_offset    "V/V"    1   LOOP_FREQ   0.0 #-0.04  
ACL_CONST_CH NF_V_offset    "V/V"    1    LOOP_FREQ   -0.19 #-0.19 

• IMC UGF seems to be ok (around 101 kHz with 25deg phase margin)

The afternoon was spent recovering the ITF:
15:30 UTC end of SUS recovery of BS (Boschi)
16:00 UTC INJ recovery: RFC, beam centering (68976 De Rossi, Lagabbe, Spinicelli)
16:30 UTC ITF recovery (#68977 Bersanetti, Boldrini, Pinto)
22:00 UTC ITF left in UPGRADING, LOCKED_ARMS_IR

After the recovery of the RFC, of the BS and of the beam centering on the inputs, we started the recovery of the lock acquisition.

We found that the demodulation phases needed to be changed significantly to allow the lock. At the DRMI stage, the phase for MICH/SRCL was tuned through a noise injection on PRCL in the 1F stage and then adjusted by checking the coherence with the 3F counterparts before the hand-off.

• DRMI_1F:
  • B2_56MHz: 1.43 -> -1.38
  • B2_6MHz: 1.1 -> 1.7
  • B2_QD2_6MHz_V/H: 2.8 -> 2.9
• DRMI_3F:
  • B2_169MHz: 3.7 -> 0.0
  • B2_18Mz: 1.7 -> 1.1
  • B2_QD2_18MHz_V/H: 2.2 -> 3.14
• DARM_IR:
  • B1p_56MHz: 4.7 -> -1.25

• STEP1:
  • B2_169MHz: 3.5 -> -0.2
  • B1p_56MHz: 4.9 -> -1.25
• STEP2:
  • B2_169MHz: 3.6 -> -0.1
  • B1p_56MHz: 4.8 -> -1.55
• STEP3:
  • B2_169MHz: 3.9 -> -0.25
  • B1p_56MHz: 4.45 -> -1.95
  • B1p_56MHz_H/V: 3.3 -> 2.9/3.1

• CARM_NULL_3F:
  • B4_6MHz: 1.55 -> -0.1
  • B1p_56MHz: 4.55 -> -1.85
  • B2_169MHz: 3.9 -> -0.25
  • B1p_56MHz_H/V: -> 2.4/2.4

While the lock acquisition up to CARM_NULL_3F is reliable, the ITF cannot maintain the lock more than a couple minutes in this state, due to an issue similar to what was observed about a month ago (68724): noise on DIFFp propagates to longitudinal signals and kills the lock in a short time. Unfortunately the attempts to tune parameters before the unlock were unsuccessful, the best way forward would likely be to disengage DIFFp while the state is stabilized. Noticeably, the phase for B2_169MHz is likely mistuned, as the coupling between MICH and PRCL is high, but we could not tune it in time before each unlock.

We leave the ITF with the arms locked.

Triggered by the low power of RFC TRA, we started to move the piezo of M15 and M16 to try to better realign it. We could optimize from 2.4 V to 2.8 V, but then we realized that yesterday at around 12 UTC the transmission crossed a maximum above 3 V, and this was in correspondance to the alignment of the north arm and a movement of the PR transversal position. 

Moreover, since after the shutdown the beam was no more centered on the ITMs, we moved PR X and Y to recenter it, with the arms locked (see attached plot).

After moving the PR the RFC TRA power was low again and by undoing the steps done at the beginning on M15 and M16 we could recover 3.2 V.

On March 23 the same grounding current injection setup was used to inject sinus lines. The scope was to observe if any non linearity is produced in hrec, which the comb injection might have hidden. 

The UTC times are:

• 11 Hz line from 17:20:00  to 17:29:00
• 21 Hz line from 17:30:30  to 17:35:30

Both injections produce a large effect in Hrec (Figure 1). A large peak is seen in hrec at the same frequency of the line. As well, a large magnetic line is produced by the injection and observed by CEB magnetometers. This effect is somehow similar to what observed when the comb was injected (see mother elog).

Some non linear effect is also seen in Hrec. 

• in the case of the 11 Hz line injection, a small (4 orders of magnitude smaller!) peak at 33 Hz (3rd harmonic) is also excited in Hrec (Figure 8). Some short excitation at 22 Hz  is also visible in hrec when the 11 Hz excitation onsets (Figure 3).
• in the case of the 21 Hz line injection, the noise seen in hrec has a broad structure with sidebands, at +-1.4Hz and about +-3.3 Hz (Figure 10). Also, a 63 Hz (3rd harmonic, as well 4 orders of magnitude smaller) is excited (Figure 9).

Switching off the line at 17:35:30 caused the ITF to unlock (Figure 7).

One reason for a non linear coupling might be that the injected noise produce an oscillating electrostatic force on the charged mirror. In the past this was done by directly injecting a common mode voltage into the actuation coils (see https://logbook.virgo-gw.eu/virgo/?r=48306 and links therein) for the purpose of measuring the mirror charge. 

Note that, in case the noise in hrec is due solely to an electrostatic force on the charged mirror we would expect a sinusoideal electrostatic force at f to cause the mirror to move at both f and 2f if the mirror is charged, and only at 2f if the mirror is not charged (as described in page 15 of https://www.mdpi.com/2075-4434/8/4/82). The result we observe here is not easily interpreted as this kind of effect.

A periodic disturbance on several INJ signals has been observed after the restart of the system. After an investigation on several possible causes, a correlation with the DC value of MC MAR angular corrections has been found. The glitches disappeared after the zeroing of all the DCs by means of the motors.

Yesterday the B1_PD{1,2}_DC offsets wee adjusted by Romain:

• 2026-04-08-19h18m30-UTC    info gouaty       SDB2_dbox_bench saved - gouaty: adjust B1_PD2 offset (rev. 269159)
  2026-04-08-19h19m45-UTC    info gouaty       SDB2_dbox_bench saved - gouaty: adjust B1_PD1 offset (rev. 269160)
  2026-04-08-19h19m48-UTC    info gouaty       'Reload config' sent to SDB2_dbox_bench

After this operation, the SDB2_LC and the SQZ_CTRL became blue in VPM interface complaining about possible missing samples

This morning we made some investigations on the rtpc1 servers

• by changing  the TOLM_PROCESSOR_LOOP delays from 25us to 28us and then restarting most of  the rtpc1's servers execpt the DaqBox ones 
• and then reconfiguring the   SDB2_MezzPD0  and the  SDB2_MezzPD1 service mezzanines 
• but without any improvments 

We put back the standard  TOLM_PROCESSOR_LOOP value (25us)  and we  continued with the investigations . We found that

•  the SDB1_FAST_SHUTTER server task order can change . To fix its order a dependancy with the SQB1_Quadrants server has been added and now this server is always excuted after the SQB1_Quadrants one . Note that the SQB1_FAST _SERVER is the server driving  the OMC dac channels 
• re-starting any ACL's server on the rtpc1, is enough to recover the correct running conditions for the 2 faulty servers

At the end, as a simple workaround, a reloadConfig  of the SDB_EDB_Tpro server allows recovering the correct running conditions without stopping any ACL's servers

Yesterday we recovered NE tower and the top stage of WE. Unfortunately Sc_WE crate was not able to transmit data to the DAQ. Investingating the issue we identify the problem: data transmission stopped as soon as the p6 DSP connected to the mirror (serial #44226) was inserted in the crate. This morning the board FPGA was reprogrammed and tested and the board has been re-inserted. WE control system seems now to be back to standard operational condition. The full suspension recovery should be now complete. We will have the final confirmation at next lock.

ITF found in UPGRADING mode and DOWN state, with SAT recovery in progress:

• Valerio was at WE working with Alain, trying to fix the missing data issue for the WE suspension;
• Paolo was closing the loops for all other suspensions.

At around 15:00 UTC, Paolo restored all suspensions (except for WE). Following this, I recovered the SNEB vertical position and closed the loops for SNEB, SDB1, and SDB2.

We then proceeded with the ITF recovery (see 'Recovery part 1' for more details).

From 18:00 UTC to 19:30 UTC, Romain performed a check of the OMC in NI single bounce, see  (see DET recovery for more details).

ITF left with Nort arm locked.

Software

At 19:30 UTC I restarted the BacnetServer because it was providing flat data.

2026/04/03:

After the DAQ recovery ( https://logbook.virgo-gw.eu/virgo/?r=68964 ), all quadrants were recovered (shutters opened and Vbias enabled), and we enabled back the photodiodes used during the lock acquisition.

2026/04/07:

We recovered the angular control and the position control of all the minitower detection benches (SDB2, SPRB, SIB2, SNEB, SWEB) as well as SDB1 (after the suspension team recovered its suspension).

2026/05/08:

DET_MAIN node recovered in SHUTTER CLOSED state after adjusting the B1s and B1_PD3 electronic offsets, and with a beam visible on the B1p camera.

Adjusted demodulation phases for the OMC lock error point (B1_PD3_f1 and B1_f1) with NI single bounce beam. For B1_PD3 the demodulation phase was changed from 1.5 to 3.4 rad (Fig.1). For B1 the demodulation phase was changed from -0.7 to +1.0 rad (Fig.2).

We also adjusted the offsets of the B1 PD1 and PD2 photodiodes.

So this afternoon there was some work on recovering the WE DSP. In the meantime we finished aligning the North arm and we locked without major problems.

During the first part of the afternoon the INJ was a bit unstable most probably due to the works outside the MC building. Once it locked in a stable way, we tried an FMODerr, whose new working point was very far from the previous one.

Once this was done we re-tested the lock of the north arm and we proceed to align the west arm. Valerio informed us that the WE MIR board was not working, but that it shouldn't prevent it from locking (in principle we only use the mirrorrs of the end test masses in LN). However, after we realigned and we tried to lock, we realized that the drift control was not properly working. Indeed only the lines in the WI mirrors were turned on, the ones on the BS and WE were not, and in this way the drift control would misalign the cavity. Paolo checked and there was some flag that was passing through the WE mirror board, so we decided to stop for the day, since we need to fix the WE board problem anyway.

At this point we decided to test the NE ALS loops. They worked fine. We stopped here for the day.

ITF found in UPGRADING Mode and DOWN State.
All times are UTC.
08:24 PR vertical and longitudinal position restored (Operator and Ruggi). NI, WI, and BS, TX, and TY restored to the values of the lock of Mar 24, 2026 00:00:00;
08:39 SWEB_SBE, and SNEB_SBE Position loops opened manually (Gouaty).
09:00 Started OMC Lock requested by Was, and Gouaty setting manual MISALIGNED State for WI with no control of WE, and NE suspensions, since the recovery of those suspensions was not started yet. Activity concluded at 09:10 due to loss of connection with all the suspension due to the stop of the SatServer, and SusDAQBridge VPM processes, caused by the SAT activity. 
09:15 SDB2_SBE, and SPRB_SBE Position loops opened manually (Gouaty).
12:05 TCS_AuxCooling_WI_FLOW flag in the DMS started oscillating between red and yellow state, the temperature (TCS_AuxCooling_Tank_TE) is rising since 09:00 UTC but it is below the threshold (see Figure 1), TCS experts informed and check of the system ongoing (Zaza). These oscillations may be related to the Air Conditioning system, Soldani is going to TCS Room to investigate the issue.
12:40 Physical restart of the chiller of the Neovan in order to restert the VPM process LaserGuardianINJ (Spinicelli).
ITF left in UPGRADING Mode and DOWN State. Recovery of SAT is still ongoing.
 

 2026-04-02

The DAQ recovery, after the computing shutdown,  started on 2026-04-02-18h28m45-UTC once the CmNames server and the VPM server were restarted and that all the expected hosts (rtpc, olserver and stol) were alive .

We restarted  fisrt all the storage and data collection parts

• raw stream on stol01 and stol02  hosts
• rawfull, rds and trend streams on stol03 host 
• the data collection parts running  on
  • the olserver52 and olserver53 hosts
  • running on the rtpcs (TolmFrameBuilder)
• operations performed between 2026-04-02-18h28m48-UTC and  2026-04-02-18h38m44-UTC 

As everything seems to run correctly , we continued by restarting all the servers involved

• in the data access part 
• in the Detector Monitoring System 
• in the Automation part 
  • The PyHVAC server was unable to retart due to
    • the missing /dev/shm/zFbsIng imput SHM
    • and later the missing  input channels  V1:HVAC_INJ_TE_OUT_SET 
    • server fully recovered the  2026-04-07-08h27m37-UTC
• in the Image readout 
• operations performed  between 2026-04-02-18h39m15-UTC and 2026-04-02-19h08m44-UTC

We left all DAQ running during 1 hour to check that everything ran smoothly, thank to the availabilty of the DMS and VIM web pages.

 Then we continued by

• reconfiguring all the Tolm devices and by restarting all the rtpc's servers without closing the loops, except for those that are automatically closed by the ACL server.
• restarting the Telescreen 's servers
• restarting the environment's  servers in the Detector's Environnent Monitoring part 
• restarting the Newtow Noise servers
• operations performed between  2026-04-02-20h10m49-UTC and  2026-04-02-21h28m35-UTC

We left all the things running during the nigth to perform a more accurate check the next day

 2026-04-03

We found that a lot of Ethernet devices (lnfs, sqz, power suppplies ) not any more recheable .  

For  DAQ/DET ethernet devices

• the DaqBoxes  were reachable 
• the SBD power supplies were not reachable.
• the SDB internal switches were reachable 
• the SWEB QD  RD482-ETH bridge was not reachable, meaning the B8_QD{1,2}  shutters and Vbias were not anymore remotely driven and monitored

Thanks to the local support, the  SBD power supplies  remote monitoring/driving were recovered . 

To recover remote monitoring/driving B8_QD{1,2},

• the SWEB_PowerUnit12 was turned off/on and fortunately it was enough to recover this device.
• as the internal Timing mezzanine was switched OFF/ON to, the SWEB bench DBoxes were reconfigured 
• to recover the correct running conditions , the SWEB rach DBoxes were reconfigured too and the Acl's SWEB_rtpc servers restarted
• operations performed between 2026-04-03-09h19m33-UTC  and 2026-04-03-09h37m46-UTC

Easter break

During the Easter break, the olserver115 became unreachable around 2026-04-03-19h34m38-UTC . The MdVim server has been restarted temporaly on the olserver118  at 2026-04-04-04h20m30-UTC.

The olserver115 host has been recovered by the IT department and the MdVim server is now running on its host since  2026-04-07-06h32m25-UTC

2025-04-07

All the Fbs servers were restarted with a new release v8r24p0 

• operations performed between 2026-04-07-15h03m20-UTC and 2026-04-07-15h08m29-UTC

This morning we restarted the injection by:

- closing the EIB suspensions 

- restoring the previousl pump currents in the Neovan diodes (5 A and 4.9 A respectively)

- restoring the previous currents to the SL diodes (29.1 A each)

- locking PMC, closing BPC and manually aligning the IMC to reach the threshold for the lock and the automatic alignment to be engaged 

There is slighlty less power than before the shutdown, both in transmission of the Neovan and the PMC (4% of power missing).

From the camera the output shape and position of the neovan seemed ok (picture 1), so we went to the laser lab to align the PMC. With a very little tilt in the vertical direction we minimized the 01 peak, however the target power was not recovered (0.75 V vs 0.78 V).

The RFC has less power in transmission but we will wait for the ITF to be aligned because it has an influence on it.

This afternoon, a few hours after the restart of all CO2 lasers, thermal camera images of both CO2 benches were acquired in order to check the shape and position of the DAS rings/CH for both WI and NI.

In Figures 1 and 2, today’s images are compared with those acquired on 2026-03-10.

No changes in the shape or position of the actuators are observed compared to previous acquisitions.

CEB (BPC, IB, BS, NI, WI, PR, SR, SDB1) and MC SAT control systems have been recovered. BPC, IB and MC have been restored on the afternoon of last Friday to allow INJ subsystem recovery. The rest of CEB tower control systems have been restored today. Tomorrow we will proceed with the recovery of NE and WE SAT control systems.