19/11/2025

~7h26 UTC : stop all WEB and NEB rotors

Test of vertical sensor of WSN: we swapped the lemo cable between WSN_vertical_pos and WSN_lateral_pos on the adc side and put them back in place:

• 7h35 :05-25 utc swaping lateral and vertical
• 7 :36 :00-15 utc swaping back to « nominal »
• Conclusion : looks like there was a lemo connector not properly plugged on the adc side. It is cured by this simple manipulation (see sensor pos plot)

Survey on WEB ( use names from wiki)

• Place jig bars under the platines and jig cylinder at the far jig position for WE(N,M,F) then WN(N,M,F). (7 :50 - 7 :52 UTC) (see Positioning_jigs_WE_WN.png)
• Move the counterweight of WN(NMF) platine around 8:54 (picture CounterWeight_WN.png) to enable the survey
• Survey for WE and WN
  • with Station 1
    • starts around 9:38 UTC and ends before 9:49 UTC. some position sensors changed when moving sphere to top of WNN rotor at gps=1447580619.
  • calibration of station 2 around 9:50 UTC
  • Measures with station 2 start at ~10:05 UTC
    • WE top, and WE near
    • move of counterweight (done just before 10:09:08 UTC 1447582166) to enable WE far measurement
    • WN near, far, top (in this order)
    • End at 10:12:03 UTC 
  • Turn WEN and WNN by 180° (ended at 10:21) (see image taken after rotation) . We noted that WNN is not completely stable on the table (some wobling).
  • Need to move again counterweight to measure WE points.
  • Start measure with station 2 and rotated rotors around10:21
    • first measured point (WEF before couterweight move)
    • WEN after couterweight move
    • WETop2
    • 10:25:14 WNN; 10:26:44 WNF ; 10:27:24 WNTop2
• Place jig bars under the platines and jig cylinder at the far jig position for WW(N,M,F) then WS(NMF) between ~10:35 UTC and 10:40 UTC
• Survey for WW and WS
  • Install and calibrate station 3 for WW survey
    • Start measurements at 11:20:58 (WWF, WWN, WWT) ended at 11:24:29
  • Install and calibrate station 4 for WW survey
    • Start measure ~11:39:22 WWN, WWT. End at 11:40:41
    • Rotate WWN by 180° (see picture of rotated rotor) between 11:40:41 and 11:4
    • Between 10:46 and 10:47, measure again WWN, then WWT2
  • Lunch break
  • 15:50 : start installation and calibrate startion 5 for WS survey at ~14:56
    • start measure WWF at 15:03:47; WWN at 15:05:35; WWT2 at 115:06:14... end at 15:07:15
    • WST at 15:08:13--> 15:09:07
    • rotate WSN --> 15:16? (rotor has now its electronic on "interior side" as the other rotors from WEB)
    • WST2 at 15:16-15:17 end of measure
• ~15:19 remove jigs from WW(N,M,F) and WS(NMF)

All rotors from WEB are now rotated by 180° compared to beginning of the day

Return to control room and restart all rotors at 2* nominal frequency @ around 16:05

To be continued on november 20.

7:00 UTC ITF found in DOWN with the following issues :

• DRMI_LOCK node stuck due to SR misalignment process malfunction. Solved by acting on SUS_SR and SUS_PR nodes (operator)
• Various error messages on different VPM processes (Masserot)
  • frame 1447574578 not send to dy_xxxxxxxx: sending queue is full
  • FdTOTProcessOne: 1447574000 nData error for V1:Daq_FbTrend100s_1000_dirRemovedGB
  • FbtBuildTrendFrame. Vector overflow at 1447438000 index=195 samplingRate=1 channel=V1:SIB2_B2_QD1_H_DC_missing serData:V1:SIB2_B2
  • etc ...

COMMISSIONING mode set (keeping LOCKED_ARMS_IR)

7:04 UTC Ncal rotor position metrology (Mours, INF)
8:15 UTC Measurements and door installation (partial) at MCB, WE, NE, CB (Fabozzi, Romboli)
9:37 UTC Neovan chiller assessment (Gherardini, Gosselin, De Rossi)


 

Since 2025-11-18-22h30m-UTC, there is regular intervals of missing data related to SUSP_Fb TolmFrameBuilder . As first trial:

• the FbmFFE and the SUSP_Fb servers were restarted, but without improvement

  • 2025-11-19-07h26m47-UTC    info process:SUSP_Fb started

  • 2025-11-19-07h48m33-UTC    info process:FbmFFE started

After some investigations, it appears that the issue was due to the Tolm packect sent by the TIMING_DBOX_WR DBox which was not reconfigured after the Timing shutdown and GPS receiver restart. To allow the reconfiguration of the TIMING_DBOX_WR, the WR_NODE need to be reseted. 

Thank to some facilities at the TolmFrameBuilder level, the faulty packet has been removed from the data collection at the SUSP_Fb level : operations performed at 2025-11-19-08h24m42-UTC

• 2025-11-19-08h24m42-UTC    info SUSP_Fb saved - masserot (rev. 264691)

• 2025-11-19-08h24m44-UTC    info 'Reload Configuration' sent to SUSP_Fb

• see this picture

After these operations, SUSP_Fb seems to run smoothly.

For the record, I have turned off this morning all the O4-related cronjobs, in particular those updating the VIM plots. The png and pdf files related to O4b, O4c and O4b+c should thus remain frozen for now on.

Online BruCo continues running but I have changed its configuration. Instead of focusing on Science segments

export ONLINE_BRUCO_LOCK_CHANNEL="V1:DQ_META_ITF_Mode"
export ONLINE_BRUCO_LOCK_MIN_INDEX=1

it will now search for LN3 segments

export ONLINE_BRUCO_LOCK_CHANNEL="V1:META_ITF_LOCK_index"
export ONLINE_BRUCO_LOCK_MIN_INDEX=135

but still in autorelock mode

#export ONLINE_BRUCO_AUTORELOCK=0
export ONLINE_BRUCO_AUTORELOCK=1

I thought that segments in autorelock mode would be cleaner and make BruCo runs more useful, but I can easily flip that switch if it is more convenient to remove that cut. Just let me know.

One more thing that happened yesterday afternoon is that the power on the B1s camera was several orders of magnitude higher than normal. This was likely a consequence of trying to recover the OMC slow shutter in the morning, one of the waveplates of the OMC must have been rotated. A check in single bounce has confirmed that most of the power on B1s is in the P instead of S polarization. Moving the north waveplate by ~10'000 steps (half-wave plate if I remember well) recovered the good polarization. I haven't done a fine tuning, and the south wave plate might be off by ~100 steps, as I had first done small steps with both waveplates to try to understand which one was moved by mistake this morning (sending two commands in a row too close together can confuse the driver and make it turn a different waveplate).

This morning Alain has recovered B1 PD1 by reconfiguring two photodiode service mezzanines, MezzPD0 and MezzPD1.

The shift was dedicated to the recovery of the ITF (see report #68188) carried out by Boldrini, Bersanetti and Was. While it is possible to reach CARM_NULL_1F, ITF_LOCK remained stuck at LOKCING_DC_READOUT due to B1 PD1 not responding via VPM. 
Activity concluded at 21:45 UTC, ITF left in CARM_NULL_1F.

This afternoon/evening we worked on the recovery of the ITF.

Over the afternoon we observed multiple unlocks in different points of the lock acquisition that we ultimately discovered being caused by a mistuned phase for B2_169MHz. Looking at the MICH_PRCL_COUPLING channel, this phase was re-tuned along the lock acquisition and CARM_NULL_1F can now be reached smoothly.

Here we found new issues in locking the OMC: a bright fringe forces to misalign the SR (TY +2 urad), which in turn introduces an imbalance in the sidebands. Without a MICH offset, this imbalance makes B1p shoot up when the DARM offset is introduced and the ITF unlocks while attempting to lock the OMC.

Introducing both a misalignment of +2 urad on SR_TY and a MICH offset of 40 produces a condition in which the OMC can be locked.

With yet another stroke of bad luck, B1_PD1 is not working and does not respond to the rearm command, preventing us from reaching DC_READOUT.

This issue cannot be solved tonight, but at least we recovered the lock acquisition up to CARM_NULL_1F. We leave the ITF in this state.

In anticipation of a possible lock of tonight for NCal investigations, the frequencies of all NCal lines and associated PCal lines have been doubled (from about 36 Hz to about 72 Hz) around 7 pm UTC.

This morning, we performed a test on the valves' opening of the DET supply and return air duct distributions in order to evaluate the impact of the noise in DET Lab.

The attached drawings show the distribution of the supply (blue) and return (red) air ducts. The position of the valves along the ducts and in the different areas is indicated by the label RP (regolatore di portata, flow regulator).
The table reports the opening of the valves (checked through  Kieback & Peter interface), which are normally operated in automatic mode, as this is their standard working configuration (unlike those in INJ, which are in manual mode).

*********************  VALVES' OPENING (automatic mode) ******************************

0% corresponds to a fully closed valve
100% corresponds to a fully open valve
****************************************************************************************

Before starting the test focusing on the valve of the Minitower A area, we performed a few actions:

• ~08:38:25 UTC – BACnetServer restarted;

• ~08:54 UTC – the supply and return fan frequencies (HVAC_FREQ_OUT/IN) were manually fixed at 27.4 Hz and 13.8 Hz, respectively, through the Kieback & Peter control interface → Forzatura Manuale CTA. These values correspond to the operating configuration, which remains unchanged between SCIENCE and ACCESS modes (as referred by Davide Soldani).

• all the supply and return valves of the four areas were set to 5% opening by switching them from automatic to manual mode. The action times are listed below:

At this point, we changed the opening of the supply and return valves of the **** Minitower A area (DET lab)***:

At the end of the test, we restored the automatic mode for the valves in each area and then for the fans, performing the actions in sequence:

Figure 1 shows the acoustic noise (ENV_EDB_MIC) inside the DET Lab during the test. Analysis will follow.

The valve opening values in automatic mode returned approximately to their initial settings, except for the New SAS area, which shows 0% on the interface (Davide Soldani has been informed)

This morning we went in chiller room to replace the filter of the chiller used to cool the head of the neovan in order to see if it has any positive effects on the temperature of the neovan and consequently on some figure of merit of PSL such has neovan output power, PSTAB residual rms noise, PMC throghtput.... 
We changed the filter but the chiller did not restart. It has the usual "flow sensor" problem that it sometimes has when it restarts. This time we could not go through it and we had to connect the filter the head to the spare chiller. 
We restarts the neovan and PSL/INJ relocked without troubles. 

The temperature of the head is lower than before (first plot attached) which is not completely explained since the measured flow is the same and the temperature setpoint of the chiller is the same. 
Anyway it slightly improved the power at the output of the neovan and more noticeable the power in transmission of the PMC. 

However it seems that there is a problem with the temperature stabilization, there is an oscillation of about 20 min (second plot attached) which can slightly be seen on the downstream power signals, IMC_TRA for example. 

Yesterday afternoon entire suspension system (BS, NI, NE, WI, WE, PR, SR, OB, MC, IB) has to be recovered due to the timing issues generated by the power outage in the DAQ room. BS IP has to be recentered due to the high DC corrections sent to F0 actuators. System was back in operation around 7 PM LT. 

ITF found in TROUBLESHOOTING at LOCKED_ARMS_IR.

07:00UTC - ITF mode set to MAINTENANCE. Below a list of activities communicated to CR (times in UTC):

• excavation and transport works for planting trees near office and central buildings (D'Andrea, external firm) - 06:45
• monthly elevator maintenance (Rossi, Baldocchi, external firm) - 07:00
• standard IMC, CEB, NEB entrance, WEB entrance cleaning (Ciardelli, Menzione, external firm) - 7:00
• TCS: chiller refill (Menzione) - 7:20
• soil seismic studies near WEB (Paolucci) - 07:40
• replacement of all 6 main service water filters, 2 ultra pure water pre-filters and Millipore Q-guard elix 70 filters (Ciardelli, Zaza) - 08:00
• DET HVAC tests (Passaquieti, Tringali)  - 09:00
• WiFi hardware plugin at WE (Berni, Cavalieri) - 10:00
• Neovan cooling filter replacement (Gosselin, De Rossi, Melo) - 10:10

Periodic tasks performed by the operator (times in UTC):

• check of IMC working point - 07:00
• lock and scan of the OMC in single bounce aborted, unable to open shutter. On-call expert notified 07:12
• check of TCS powers - 07:45
• check of NI, WI thermal camera reference - 08:00

Relock (times in UTC):
07:50 - Recovered SQB2 bench vertical position, rearmed Vbias and opened shutter for B1p_QD1.
09:15 - 11:45 - Ballardin and I entered CEB to swap OMC slow shutter driver to a  working channel in order to regain control of it, as instructed by Was. Despite swapping channels and the whole AG-UC8 controller unit with a spare (current configuration in figs. 1 and 2), we remain unable to move the shutter (entry #68183).
12:43 - ITF back in PREPARE_SCIENCE mode. After cross alignment by Zaza ITF is able to lock CARM_NULL_3F for a couple of minutes.
13:30 - Restored SDB1_B5_QD2 offsets to their dark fringe values after unsuccesful OMC scan attempt.
13:40 - ITF parked in LOCKED_ARMS_IR_ALS as requested by Masserot from remote in order to restore ALS power inside arm cavities. B1p QD1,2 Vbias rearmed and shutter opened.
13:57 - ITF relocked to CARM_NULL_1F.

The OMC slow shutter did not want to open this morning. 

Figure 1. Looking at the OMC temperature band passed at 1.7kHz (fast RMS) and band passed at 666Hz (slow RMS), the electrical signal to move the translation stage of the shutter is no longer being sent. Normally there is a cross talk between the sent voltage to make the motor steps, and the temperature sensor. The last successful commands to move the shutter where on November 17 before 6 UTC, before the electrical problems at the site. Afterwards there have been many commands sent on Nov 17 in the afternoon by the automation, which did not have an effect. 

First the cable of the OMC slow shutter was moved from channel 2 axis 1 to channel 4 axis 1 around 9:15 UTC. This did not change the situation, there was still no cross talk visible on the OMC temperature. Trying to rotate the slow shutter of the Faraday isolator showed that the driver does send commands on channel 4 axis 2. Then the cables of the Faraday isolator and the OMC slow shutter were swapped around 9:45 UTC, so that the OMC slow shutter is now on channel 4 axis 2. After that there started to be electrical signals sent to the OMC slow shutter visible in the OMC temperature, but the shutter is still not moving and not letting light go through to the OMC.  

We then swapped back the OMC shutter and FI shutter cables around again. And looking at the position of the FI shutter it was also not moving, even though the electrical signals were being sent to it.

Channel 2 axis 1 and Channel 4 axis 1 appear to not work on the Agilis driver, FT4ZNKQD. Looking back into the logbook that was already known. Next time we should put some tape on the broken channels so it is more evident. The whole driver was also not moving the rotators, even though the cross talk of the driving signal was visible. Either the driver is fully not functional, or it needed more actions to recover, like turning it off and on. We haven't tested which one of the two hypothesis is true.

The driver was replaced by a spare, FT6G3NUZ. This has solved the problem, both the FI shutter and the OMC shutter started to move again. The cabling has not been updated, so now the OMC slow shutter is connected to channel 4 axis 1, instead of channel 2 axis 1. The buttons in VPM have been updated, as well as DET_MAIN.py and ITF_LOCK.py to use the new channel.

After that I have also check that the OMC lock works, and update the demodulation phase of B1 PD3 and B1 by 0.5 radian, so that all of the signal is on the I quadrature, instead of half on I and half on Q.

2025-11-17-05h24m39-UTC>INFO...-Received move_rel message from DET_MAIN
2025-11-17-05h24m39-UTC>INFO...-Parsed message*: [2, 1, 45000, 50]
2025-11-17-05h24m39-UTC>INFO...-setting channel 2 axis 1 to 45000 at speed 50
2025-11-17-05h24m39-UTC>INFO...-writing position -103695199
2025-11-17-05h56m16-UTC>INFO...-Received move_rel message from DET_MAIN
2025-11-17-05h56m16-UTC>INFO...-Parsed message*: [2, 1, -60000, 50]
2025-11-17-05h56m16-UTC>INFO...-setting channel 2 axis 1 to -60000 at speed 50

2025-11-17-13h07m18-UTC>INFO...-Received move_rel message from DET_MAIN
2025-11-17-13h07m18-UTC>INFO...-Parsed message*: [2, 1, -60000, 50]
2025-11-17-13h07m18-UTC>INFO...-setting channel 2 axis 1 to -60000 at speed 50
2025-11-17-13h07m18-UTC>INFO...-writing position -103815199
2025-11-17-13h09m15-UTC>INFO...-Received move_rel message from DET_MAIN
2025-11-17-13h09m15-UTC>INFO...-Parsed message*: [2, 1, -60000, 50]
2025-11-17-13h09m15-UTC>INFO...-setting channel 2 axis 1 to -60000 at speed 50
2025-11-17-13h09m15-UTC>INFO...-writing position -103875199
2025-11-17-13h11m11-UTC>INFO...-Received move_rel message from DET_MAIN
2025-11-17-13h11m11-UTC>INFO...-Parsed message*: [2, 1, -60000, 50]
2025-11-17-13h11m11-UTC>INFO...-setting channel 2 axis 1 to -60000 at speed 50
2025-11-17-13h11m11-UTC>INFO...-writing position -103935199

The following report has been submitted to the On-call interface.

On-call events -> Interferometer Sensing & Control

Title: ALS recovery

Author(s): derossi

Details:

Once SNEB and SWEB benches have been brought in place, while at the NEB the green beam was fine, on SWEB it looked completely misaligned (plot 1).

On the North the beating was recovered without changing anything (there is just less signal, we will run the automatic alignment to recover it as soon as possible), see plot 2. 

On the West the alignment in reflection was recovered by moving the tilt (plot 3), however there is still less signal on the PSD on SWEB and on the beating on the EIB.

Moreover, when trying to relock, we realized that also the demodulation phase was wrong (plot 4).

Many thanks for Diego and Michal's help.

* Note that any files attached to this report are available in the On-call interface.

The PCal were switched on yesterday around 13h30 (NE) and 14h30 (WE) UTC.  (entry from yesterday but not posted).

The power supply for the Timing rack located in the DAQ was restored thanks to F.Berni and M.D'Andrea

The first part of the shift was dedicated to the recovery of the DAQ Timing (see report #68173) and the Reset/recovery of the Suspension Master Board, which required the restart of all the DSP.
Suspension recovery completed at 17:55 UTC, with the help of Was it was possible to unstuck DET_MAIN and after a brief re-alignment of the cavities the ITF was back in LOCKED_ARMS_IR at 19:08 UTC.
While it was possible to reach LOCKED_ARMS_IR_ALS, the ITF started to behave erratically while reaching LOCKED_ARMS_BEATING, with strong oscillations of NE and WE suspensions and with B7/B8 flashing with cavities unlocked.
De Rossi investigated the issue, which seems to be connected to the alignment of the End Benches. 
Recovery in progress.

ITF found locked in LOW_NOISE_3 with SCIENCE mode (Autoscience OFF). BNS Range ~53 Mpc. ITF subsystems recovery from bad weather (times in UTC):
05:56 - unlocked due to bad weather (Wind and Sea Activity flags red on DMS) like tonight, ITF unable to hold CARM_NULL_1F for the duration of lock timer since. BAD_WEATHER mode set at 07:43 after multiple unlocks due to thunderstorm on site (fig. 1).
08:06 - ITF parked in DOWN state with all minitowers (both SBE and LC loops open) as agreed with Masserot, Boldrini and Bersanetti.
08:50 - 09:17 - Berni, Boldrini and D'Andrea entered DAQ room to investigate what caused all timing signals to disappear under request by Masserot. A circuit breaker had tripped and disconnected the whole timing rack. waiting for recovery of GPS and reset of master boards by Boschi.
09:27 - BS_F7 loops and guardians correctly closed, they had opened after unlock at 06:47
09:54 - Gosselin manually unlocked PMC in order to allow recovery of rtpc servers by Masserot and Viret. ITF in TROUBLESHOOTING starting from 10:17.
12:57 - Masserot finished recovering Timing and minitowers control loops. SQZ and SDB1 benches recovered at 13:20. Boschi and Piendibene entered CEB to perform master board reset.
13:24 - Restarted ITF_LOCK and olwin server under guidance from Ballardin.
ITF left unlocked.

Due to a  thunderstorm, the Timing rack was switched off : no more Timing signals available for th ITF.  From the DAQ point of view the recovery was complete around 13H-UTC

All the TOLM devices were reconfigured : ADC7674, DBox and TOLM-PCIe mezzanine . 

To allows theses operations

• all the loops were openned and all the Acl tasks were stopped .
• The Acl servers have been restarted with theirs standard configurations
  • only the SIB2, SPRB, SDB2, SNEB and SWEB LC and SBE loops have been closed
  • for the EIB, LSC_Etalon_Acl, the  {CEB,NEB,WEB}_ALS_BPC none loops are closed.
  • for the TCS_CHROCC_PR the last DAC value has been restored , to be check by the expert

I received DMS alerts about the Pcal power this morning at 8h16 and again 8h26 UTC. The Pcal looked off from the VIM plots; there were errors messages in the Acl processes about missing/delayed samples. And Alain confirmed that there has been a strom and that the timing has probably been cut.

At 8h30 UTC, I have stopped the laser Pcal (open loop and disable the power from Acl PCal_fast, and switched off the pump diodes of the laser from PCal_WE,NE_Ampli).  We will switch them on when the timing will be back.

ITF found in LOW_NOISE_3 and in Science Mode. It kept the lock for the whole shift.
ITF left locked.

Guard tour (UTC)
17:12 - 17:48
19:10 - 19:46