ITF found in LOW_NOISE_3 and SCIENCE mode (Autoscience ON); BNS Range ~54 Mpc. Manually unlocked at 15:06UTC for planned EDB OMC HOM noise investigation at LN2 in COMMISSIONING mode.
ITF achieved LOW_NOISE_2 at 15:44UTC. EDB Picomotor process found dead on VPM (connection to picomt13 failed). Fixed by manually restarting the controller in DET lab and the EDB_PICO process on VPM at 16:36UTC thanks to timely, pivotal intervention by Spinicelli.
ITF back to LN3 in SCIENCE at 19:34UTC after end of HOM study.

Guard tour (UTC)
17:57 -> 18:35
20:53 ->

EnvMoni
Lights in MCB_TUNNEL and CEB_HALL show yellow warning on DMS (fig 1). During intervention on EDB picomotors lights could be seen underneath the hanging plastic screens around SA towers in central building.

Other
17:49UTC TelescreenLeft on VPM reports critical alert: wrong or no stream declaration for '1500WCleanRoom1'

Today I have locked the EDB OMC again on HOM in LN2. This time without realigning the OMC after the first initial realignment. There is no coherence with any of the HOM that I have tried. 

Initially the EDB picomotor process could not be reached. I must have forgotten to switch off the server last week, and it lost connection on Sunday. Piernicola and Bernardo turned on and off the picomotor manualy, which restarted the connection.

aligned EDB OMC on 56MHz USB, first to maximize the power, then ~50
picomotorse in horizontal to reduce the jitter peaks

16:47 UTC (5min) - locked on 56MHz USB

when going through carrier TEM00 there is ~0.1 mW on it. The 6MHz TEM00 are much smaller, around 0.02mW.

Adjusted demodulation phase to put all the signal in the i quadrature, this must have increased the gain by factor 3 or 4 (and the calibration)

17:09 UTC (5min) - locked on carrier TEM00 (very noisy and low power), not sure if the locked actualy worked or just stabilized in temperature around the mode

Reduced the gain by factor 2 (and calibration of error signal, still
probably wrong by factor 2) as the loop was oscillating when locking
on TEM00 56MHz LSB

17:19 UTC (5min) - locked on 56MHz LSB

order 1 mode of 56MHz sideband has about 0.05mW of power, 10 times less than the TEM00

17:30 UTC (5min) locked on carrier order 2 mode, mostly horizontal

17:40 UTC (5min) locked on carrier order 3 mode, 1-vertical, 2-horizontal

17:50 UTC (5min) locked on carrier order 4 mode, 4-vertical, 0-horizontal

17:59 UTC (5min) locked on carrier order 5 mode, 0-vertical, 5-horizontal

18:07 UTC (5min) locked on carrier order 6 mode, 1-vertical, 5-horizontal

18:16 UTC (5min) locked on carrier order 7 mode, 1-vertical, 6-horizontal

18:26 UTC (5min) locked on carrier order 8 mode, 1 vertical, 7 horizontal

18:35 UTC (5min) locked on carrier order 9 mode, 9 vertical, 0 horizontal

18:46 UTC (5min) locked on carrier order 9 mode, 1 vertical, 8 horizontal

18:56 UTC (5min) locked on carrier order 10 mode, 0 vertical, 10 horizontal

19:06 UTC (5min) locked on 56MHz USB TEM00, alignment is still good
 

In the last few days we had (again) three occurrences of a rather obscure problem, where the management of ITF_LOCK by ITF_CONDITIONS (Autorelock & Autoscience node) conflicts with some leftover request from the last management from CALI, in these cases the last requested state during Monday's calibration shift, LOCKED_PRWI.

These requests happen after an unlock, as then is when we request again something to the node, and they are not the cause of the unlock, as stated.

Looking at the unlock from this morning, the SDB1 fast shutter closes for some reason at 06:10:09.93 UTC; the signals based on SDB* are immediately cut off, while the other ones are alive for a bit more time. Everything is supposedly locked, and both the automation and the fast global control are up at the time. The fast LSC_ENABLE flag goes down at 06:10:10.03 UTC, so apparently not as a consequence of a cm command. ITF_LOCK enters the DOWN state at 06:10:11 UTC sharp from the DAQ/SMS point of view, but actually at 06:10:13.00 UTC according to the logs.

Looking at the zoomed plots of the event, we can see that the fast shutter behaviour was triggered by some glitch in either DARM or DARM/end mirrors correction signals, before the B1 saturation and the shutter closing.

From the automation point of view, the weird request is seen by ITF_LOCK and I suspect it happens more frequently than expected, but we see it if happens as the last one, so the one which sticks, while the first (the good one, in this case) is overwritten:

2025-07-03T06:10:13.008Z ITF_LOCK JUMP target: DOWN
2025-07-03T06:10:13.008Z ITF_LOCK [LOW_NOISE_3.exit]
2025-07-03T06:10:13.008Z ITF_LOCK STALLED
2025-07-03T06:10:13.537Z ITF_LOCK JUMP: LOW_NOISE_3->DOWN
2025-07-03T06:10:13.537Z ITF_LOCK calculating path: DOWN->LOW_NOISE_3
2025-07-03T06:10:13.544Z ITF_LOCK new target: FMODERR_TUNED
2025-07-03T06:10:13.554Z ITF_LOCK executing state: DOWN (1)
2025-07-03T06:10:13.555Z ITF_LOCK [DOWN.enter]  
2025-07-03T06:10:13.575Z USERINPUT REQUEST: olserver52.virgo.infn.it virgorun  old=LOW_NOISE_3 new=LOW_NOISE_3
2025-07-03T06:10:13.585Z USERINPUT REQUEST: olserver52.virgo.infn.it virgod  old=LOW_NOISE_3 new=LOCKED_PRWI
2025-07-03T06:10:13.659Z ITF_LOCK in DOWN main
2025-07-03T06:10:13.674Z ITF_LOCK ['ITF_LOCK', 'DRMI_3F', 'CARM_SET_STEP1', 'CARM_SET_STEP2', 'CARM_SET_STEP3', 'CARM_NULL_3F', 'CARM_NULL_1F', 'DITHERING', 'SHUTTER', 'DC_READOUT', 'LOW_NOISE_1', 'LOW_NOISE_2', 'LOW_NOISE_3']
2025-07-03T06:10:14.008Z ITF_LOCK REQUEST: LOCKED_PRWI

It is also seen by CALI, and this is what should not happen, as in the DOWN state of CALI we clearly state to release ITF_LOCK from management (nodes.release(), and apparently it does, from the GUI point of view):

2025-07-03-06h10m13-UTC>INFO...-Unstalling ITF_LOCK
2025-07-03-06h10m13-UTC>WARNING-[DOWN.run] ezca: V1:GRD-ITF_LOCK_REQUEST => LOCKED_PRWI

In some cases, like yesterday, this happens with CALI complaining about connections errors:

2025-07-02-11h52m19-UTC>INFO...-Memory Used increase, cur. 2411364.00(KB), inc. 8.00(KB)
2025-07-02-12h56m41-UTC>INFO...-Unstalling ITF_LOCK
2025-07-02-12h56m41-UTC>WARNING-[DOWN.run] ezca: V1:GRD-ITF_LOCK_REQUEST => LOCKED_PRWI
2025-07-02-12h56m41-UTC>WARNING-[DOWN.run] USERMSG 0: EZCA CONNECTION ERROR: Any Other Error: Invalid REQUEST: LOCKED_PRWI
2025-07-02-12h59m41-UTC>INFO...-Memory Used increase, cur. 2411368.00(KB), inc. 4.00(KB)
2025-07-02-12h59m43-UTC>INFO...-Memory Used increase, cur. 2411520.00(KB), inc. 152.00(KB)

I will look more at the problem as soon as possible, one possibility is how the "unstalling" of nodes work, as it could be the actor that tries to re-instate a management that doesn't exist anymore, but this is still unclear. Another possibility is that we do the un-management wrong in some way (this is the only place where we un-manage nodes).

also the B4 12MHz show a correlation with the range. Moreover also the coupling of frequency noise has a strong correlation with the BNS, but in a strange way... the higher is the coupling the better is the range (figure 2 and 3). The origin of the trend has not be found yet

Last unlock at GPS time 1435519088 is due to Metatron glitch requesting LOCKED_PRWI state, same as this morning's only unlock at 06:10UTC.

This morning the ITF unlocked once at 6:10UTC with this request to ITF_LOCK:

2025-07-03T06:10:13.585Z USERINPUT REQUEST: olserver52.virgo.infn.it virgod old=LOW_NOISE_3 new=LOCKED_PRWI

???, it relocked at the first attempt and science mode started at 7:03UTC, science mode stopped one minute at 8:12UTC to change the NI etalon set pint.

ISC
- 8:12UTC: NI etalon set from 20.24 to 20.34, 54000 sec ramp time.

Yesterday we observed an other strange increase of the Range

the interesting thing is that is correlated with the inverse of the power on B1p, as for the event of the 20th of Jun. See figure 1 and 2. this time no correlation with a kick on SR tx.

to be understood.

ITF found in LN3. SCIENCE mode. It kept the lock for the whole shift.

Guard tour (UTC):
20:59 - 21:35
23:10 - 23:40
01:07 - 01:43
03:11 - 03:57

ITF found in FMODERR_TUNED after recent unlock, lock acquisition in progress (Autoscience ON).

• Unlocked at 13:16UTC (pdf1) and again at 13:28UTC (pdf2) after reaching CARM_NULL_1F due to diverging SSFS correction.
• Relocked in LN3 at 14:21UTC, back in SCIENCE mode;  BNS Range ~49Mpc.
• Unlocked at 14:36UTC (pdf 3) again due to diverging SSFS correction; INJ on-call (Melo) notified.
• Automation stuck waiting for DET_MAIN in unknown state; fixed (as usual) by resetting node to INIT at 14:57UTC. Relocked at 15:30UTC.
• Unlocked at 16:01UTC (pdf 4) due to end TM (MIR or MAR) correction saturation possibly caused by high wind activity; speeds in excess of 50 km/h on site (as seen from DMS>WindActivity and MeteoStations red flags.)
• Relocked at 16:44UTC on first attempt despite above threshold wind speeds.
• Unlocked at 19:17UTC (pdf 5) due to multiple causes with yellow flag under DMS>SeaActivity>ENV_CEB_SEIS_V.
• Relocked at 20:04UTC on first attempt. Horizontal and Vertical QD1 B5 Galvo loops opened at 19:28 UTC, manually closed via VPM at 20:06UTC (fig. 1).

Guard tour (UTC)
18:06 -> 18:43
20:59 -> 

After the WEb UPS intervention, the Sc_WE glitch debugging channels have changed, meaning that the DSP codes are not exactly the same as before the UPS intervention

• before 2025-07-01-05h00m00s-UTC
• after 2025-07-01-16h00m00s-UTC
  • Sc_WE_txAA is not CAL_WE_MIR_Z_CORR delayed
  • Sc_WE_ErrPost is not CAL_WE_MIR_Z_NOISE delayed

At my arrival, locking acquisition was in progress. ITF had already experienced several unlocks

• 23:28UTC, back in Science

ALS
21:29UTC, I called the expert because the unlocks appeared to be due to low power in the WE ALS system
M. Gosselin realigned the green on WE. After that, D.Bersanetti tuned the thresholds in the automation.

QPD
22:28UTC I rearmed vBias of B1p_QD1 and B1p_QD2 

Guard Tours (UTC):
21:05 - 21:47
23:03 - 23:42
01:03 - 01:43
03:11 - 03:48

Afterwards, I had to change the thresholds in the automation:

lock_thr = 0.5
dc_trig_refl_hi = -0.5
dc_trig_refl_lo = -0.6

plot attached 

The following report has been submitted to the On-call interface.

On-call events -> Interferometer Sensing & Control

Title: ALS warm realignement

Author(s): gosselin

Details:

I have been called by Beatrice because there were some trouble to go through the lock acquisition. The West arm was unlocking.
I checked and see that there was some power missing on the signal WARM_BEAT_DC.
I first try to realign the beam in transmission, ie into the fiber on EIB but the improvement was very little.
I then reliagned the beam in the arm with the galvo on SWEB. I slightly changed the set points of the tilt X and tilt Y and recover a much greater signal in transmission.
The lock acquisition went through the green lock but eventually unlocked at CARM NULL 1F.

* Note that any files attached to this report are available in the On-call interface.

The activity started in the WE building with the NCal power supplies switched off at 6:21 UTC. Then, the WEN (box 08) and WWN NCal have been rotated by 180% to study the mirror/NCal vertical offset (see pictures). The power supplies have been turned back on at 6:39 UTC to check that the NCals were still working as usual. They have been switched off at 6:50 UTC.

Then the activity switched to the NE building.

The rotor R-10 which was removed on Jun 6 (see logbook entry) has been put back as NNN (see picture) after fixing it at IPHC. The rotor R-17 has been reinstalling on the west setup as NWN (reusing some of the old NEF rotor) and the cabling completed (see picture of the two DAQboxes used by the NCals). The East setup has also been completed with the NEN installed on the left part (see picture). The position of the suspended support has been adjusted to better than 0.1 mm.

The configurations of the SNEB_dbox_rack and NEB_NCal processes have been updated (and restarted) to support these new NCals. To get NEB_NCal working properly, the TCS HWS DAC channel (which is currently unused) has been commented out.

Finally, the new NWN and NEN rotors have been successfully tested for a short time around 16:43 UTC. They have been left off.

ITF found DOWN in MAINTENANCE mode. Recovery of WE concluded at 16:00 UTC. At 15:33 UTC I set COMMISSIONING mode.
Ncal activity at WE performed by Mours and Masserot up to 17:00 UTC.
PREPARE_SCIENCE set at 16:52 UTC.
ITF relocket at LN3 at 17:40 UTC. SCIENCE mode set. ATOSCIENCE engaged.

At 19:39 UTC ITF unlocked due to an unlock of IMC. DET Metatron node stuck in unknown state. Properly recovered. RFC took 10 minutes to relock. Bersanetti tuned again the gain of W ALS. ITF Relocking in progress after a realignment of North cavity and the usual cross-alignment in ACQUIRE_DRMI.

Guard tour (UTC):
17:55 - 18:35

Electricity
(01-07-2025 14:30 - 01-07-2025 15:00) On site
Status: Ended
Description: UPS_CB failure
Actions undertaken: Triggered by the DMS red flag of UPS_CB, that was in failure, I tried in vain to contact electricity responsible (OnCall not present in the schedule). So I contacted Andreazzoli to let him check the system. Upon he came on site, we realized that the UPS3 of CB was restored by someone else (who didn't communicated anything to the control room).

WE PCal switched on by Suzanne around 13h UTC

LASER

1435409988  01 Jul 2025 12:59:30 UTC  (01 Jul 2025 14:59:30 CEST) SWITCH ON

1435410007  01 Jul 2025 12:59:49 UTC  (01 Jul 2025 14:59:49 CEST) KEY ON

1435410084  01 Jul 2025 13:01:06 UTC  (01 Jul 2025 15:01:06 CEST) PUMP ON

1435410113  01 Jul 2025 13:01:35 UTC  (01 Jul 2025 15:01:35 CEST) REMOTE ENABLED

POWER SUPPLY

1435410166  01 Jul 2025 13:02:28 UTC  (01 Jul 2025 15:02:28 CEST) POWER SUPPLY ON

Then, at 13:08:00 UTC : enabled the laser from VPM and the control loops.

We then saw that there was extra-noise (see figure), and we realized that the LED flashing the IRIG-B in front of Tx_PD1 was enabled (red flag in the DMS, thanks Nicola for warning).  SInce WE_Tx_PD1 is in loop, the noise was seen on Tx_PD2 and the Rx sphere instead.

At 16h28UTC, we disabled the LED: PCal noise level back to normal.

The following report has been submitted to the On-call interface.

On-call events -> Electricity

Title: UPS3 problem

Author(s): andreazzoli

Details:

UPS3 power supply problem. Batteries in discharge from 14:20 to 14:40 ( UTC) . Arriving in site i confronted the operator.
The problem had been solved. Monitored ups operation by DMS. Batteries correctly charging at 54% (17:42 local time) and increasing

* Note that any files attached to this report are available in the On-call interface.

switch off

•  06:50 UTC open PSTAB and BPC at WEB
•  06:50 UTC switch-off 2W ampli
•  06:51 UTC Switch off PPLN crystal (temperature 117.63C)
•  06:52 UTC SOA current: from 69.5mA lowered to 0 and switch off (current unplugged at 06h53)
•  06:58 UTC unplug the corrections to the RF amplifier of the AOM (PSTAB cable 8 and VCO cable 6 unplugged)
•  07:03 UTC VCO alim L1.3 off 
•  07:10 UTC switch off the alim of the RF amplifier + photodiodes box ALS DCG L1 and 07:11 UTC L2+L3
•  07:13 UTC power supply alim L1, L2+L3 unplugged

switch on

• 13:10 UTC power supplies alim L1, L2+L3
• 13:10 UTC switch on the alim of the RF amplifier + photodiodes box ALS DCG L1, L2, L3 
• 13:11 UTC VCO alim L1.3
• 13:11 UTC plug the corrections to the RF amplifier of the AOM (PSTAB in can 8 e VCO in  can 6 )
• 13:12 UTC power supplies SOA and crystal without switch on 
• 13:12 UTC PPLN crystal oven and SOA switched on (current increased to 69.5mA)
• 13:13 UTC 2W amplifier on 
• 13:24 UTC PSTAB and BPC closed 

Relock of the green at around 15:05 UTC (we slightly changed the BPC WEB  tilt x setpoint from -0.001 to -0.135 to improve the alignment and tuned the thresholds in ALS_WARM.ini).

After some unlocks possibly due to the IMC cavity during the last few days, we decided to check the status of the IMC control and working point.

After experiencing some unlocks, possibly due to the IMC cavity, over the past few days, we decided to check the status of the IMC control and working point.

We started by measuring the IMC OLTF.  Compared to what we had few month ago, we measured a higher phase margin (~30⁰) but with a lower UGF (~100 kHz).

We then decided to increase the gain (-1 dB on the attenuator) in order to raise the UGF, and we eventually obtained ~125 kHz with a phase margin of ~23⁰ (see fig. 1).

We then checked the angular working point. Only minor corrections were needed, except for FF_V (+0.4 → +0.58). The new values have been applied and stored in the ACL.

Today, around 14h30 UTC, I have restarted on olserver117 (16 CPU) the process MdDQ which was running on olserver119 (4 CPU).
MdDQ starts each day at 6h30 the full reprocessing of the VIM plots of the previous day, so that they all start at 00h00 UTC (for the time plots) or are done at the max BNS range of the day (for the FFT plots). But, the number of scripts used in VIM has increased and this reprocessing was taking 100% of the 4 CPU and all the RAM of the olserver119 machine.
(see http://ganglia.virgo.infn.it/?r=month&cs=&ce=&m=load_one&c=New+Olservers&h=olserver119.virgo.infn.it)
and many scripts were killed during this daily reprocessing (as can be seen in /opt/w3/vim/util/reprocess_log.txt).
Thus, I have restarted MdDQ on olserver117, which has 16 CPU and much more memory. We will see if it solves the problem.

* PRELIMINARY *

Taking advantage of the UPS work at WEB (mains shutdown), we can compare the seismic, acoustic and magnetic noise levels of WEB (in purple all OFF, in blue standard conditions).

There is a huge difference in the triaxial magnetic fluxgate near the vacuum tank (ENV_WE_MAG_x), it seems that we have reached the sensor noise floor. To be further investigated.

Another interesting difference is the ALS microphone noise level (ENV_WE_ALS_MIC) for frequencies above 2 kHz.

HVAC was left always ON

* MORE DETAILED ANALYSIS WILL FOLLOW *

This morning around 12:40 UTC I realigned the MZI in SQZ box to recover some contrast, see picture.

I took the chance to check the optical power of the green beams:

- before MZI: 49 mW

- SQZ box output for FC: 32 mW

- after AOM first pass: 19 mW

- after AOM second pass: 12 mW

At  07:47:17 UTC, both ETM ring heaters were switched off.

At 07.59:44 UTC, Maria switched off the WE RH power supply.

========

At  12:38:09 UTC, WE RH was switched ON.

At  12:38:59 UTC, NE RH was switched ON.