After the WEb UPS intervention, the Sc_WE glitch debugging channels have changed, meaning that the DSP codes are not exactly the same as before the UPS intervention
At my arrival, locking acquisition was in progress. ITF had already experienced several unlocks
ALS
21:29UTC, I called the expert because the unlocks appeared to be due to low power in the WE ALS system
M. Gosselin realigned the green on WE. After that, D.Bersanetti tuned the thresholds in the automation.
QPD
22:28UTC I rearmed vBias of B1p_QD1 and B1p_QD2
Guard Tours (UTC):
21:05 - 21:47
23:03 - 23:42
01:03 - 01:43
03:11 - 03:48
Afterwards, I had to change the thresholds in the automation:
lock_thr = 0.5
dc_trig_refl_hi = -0.5
dc_trig_refl_lo = -0.6
plot attached
The following report has been submitted to the On-call interface.
On-call events -> Interferometer Sensing & Control
Title: ALS warm realignement
Author(s): gosselin
| Called at: 23:15, 01-07-2025, by: Montanari |
| Remote intervention: Started: 23:15, 01-07-2025; Ended: 00:30, 02-07-2025 |
| On-site intervention: Started: ; Ended: |
| Status: Resolved |
| Operator when issue resolved: Montanari |
Details:
I have been called by Beatrice because there were some trouble to go through the lock acquisition. The West arm was unlocking.
I checked and see that there was some power missing on the signal WARM_BEAT_DC.
I first try to realign the beam in transmission, ie into the fiber on EIB but the improvement was very little.
I then reliagned the beam in the arm with the galvo on SWEB. I slightly changed the set points of the tilt X and tilt Y and recover a much greater signal in transmission.
The lock acquisition went through the green lock but eventually unlocked at CARM NULL 1F.
* Note that any files attached to this report are available in the On-call interface.
plot attached
Afterwards, I had to change the thresholds in the automation:
lock_thr = 0.5
dc_trig_refl_hi = -0.5
dc_trig_refl_lo = -0.6
The activity started in the WE building with the NCal power supplies switched off at 6:21 UTC. Then, the WEN (box 08) and WWN NCal have been rotated by 180% to study the mirror/NCal vertical offset (see pictures). The power supplies have been turned back on at 6:39 UTC to check that the NCals were still working as usual. They have been switched off at 6:50 UTC.
Then the activity switched to the NE building.
The rotor R-10 which was removed on Jun 6 (see logbook entry) has been put back as NNN (see picture) after fixing it at IPHC. The rotor R-17 has been reinstalling on the west setup as NWN (reusing some of the old NEF rotor) and the cabling completed (see picture of the two DAQboxes used by the NCals). The East setup has also been completed with the NEN installed on the left part (see picture). The position of the suspended support has been adjusted to better than 0.1 mm.
The configurations of the SNEB_dbox_rack and NEB_NCal processes have been updated (and restarted) to support these new NCals. To get NEB_NCal working properly, the TCS HWS DAC channel (which is currently unused) has been commented out.
Finally, the new NWN and NEN rotors have been successfully tested for a short time around 16:43 UTC. They have been left off.
ITF found DOWN in MAINTENANCE mode. Recovery of WE concluded at 16:00 UTC. At 15:33 UTC I set COMMISSIONING mode.
Ncal activity at WE performed by Mours and Masserot up to 17:00 UTC.
PREPARE_SCIENCE set at 16:52 UTC.
ITF relocket at LN3 at 17:40 UTC. SCIENCE mode set. ATOSCIENCE engaged.
At 19:39 UTC ITF unlocked due to an unlock of IMC. DET Metatron node stuck in unknown state. Properly recovered. RFC took 10 minutes to relock. Bersanetti tuned again the gain of W ALS. ITF Relocking in progress after a realignment of North cavity and the usual cross-alignment in ACQUIRE_DRMI.
Guard tour (UTC):
17:55 - 18:35
Electricity
(01-07-2025 14:30 - 01-07-2025 15:00) On site
Status: Ended
Description: UPS_CB failure
Actions undertaken: Triggered by the DMS red flag of UPS_CB, that was in failure, I tried in vain to contact electricity responsible (OnCall not present in the schedule). So I contacted Andreazzoli to let him check the system. Upon he came on site, we realized that the UPS3 of CB was restored by someone else (who didn't communicated anything to the control room).
WE PCal switched on by Suzanne around 13h UTC
LASER
1435409988 01 Jul 2025 12:59:30 UTC (01 Jul 2025 14:59:30 CEST) SWITCH ON
1435410007 01 Jul 2025 12:59:49 UTC (01 Jul 2025 14:59:49 CEST) KEY ON
1435410084 01 Jul 2025 13:01:06 UTC (01 Jul 2025 15:01:06 CEST) PUMP ON
1435410113 01 Jul 2025 13:01:35 UTC (01 Jul 2025 15:01:35 CEST) REMOTE ENABLED
POWER SUPPLY
1435410166 01 Jul 2025 13:02:28 UTC (01 Jul 2025 15:02:28 CEST) POWER SUPPLY ON
Then, at 13:08:00 UTC : enabled the laser from VPM and the control loops.
We then saw that there was extra-noise (see figure), and we realized that the LED flashing the IRIG-B in front of Tx_PD1 was enabled (red flag in the DMS, thanks Nicola for warning). SInce WE_Tx_PD1 is in loop, the noise was seen on Tx_PD2 and the Rx sphere instead.
At 16h28UTC, we disabled the LED: PCal noise level back to normal.
The following report has been submitted to the On-call interface.
On-call events -> Electricity
Title: UPS3 problem
Author(s): andreazzoli
| Called at: 16:24, 01-07-2025, by: Menzione |
| Remote intervention: Started: ; Ended: |
| On-site intervention: Started: 16:58, 01-07-2025; Ended: 17:45, 01-07-2025 |
| Status: Resolved |
| Operator when issue resolved: Menzione |
Details:
UPS3 power supply problem. Batteries in discharge from 14:20 to 14:40 ( UTC) . Arriving in site i confronted the operator.
The problem had been solved. Monitored ups operation by DMS. Batteries correctly charging at 54% (17:42 local time) and increasing
* Note that any files attached to this report are available in the On-call interface.
switch off
switch on
Relock of the green at around 15:05 UTC (we slightly changed the BPC WEB tilt x setpoint from -0.001 to -0.135 to improve the alignment and tuned the thresholds in ALS_WARM.ini).
After some unlocks possibly due to the IMC cavity during the last few days, we decided to check the status of the IMC control and working point.
After experiencing some unlocks, possibly due to the IMC cavity, over the past few days, we decided to check the status of the IMC control and working point.
We started by measuring the IMC OLTF. Compared to what we had few month ago, we measured a higher phase margin (~30⁰) but with a lower UGF (~100 kHz).
We then decided to increase the gain (-1 dB on the attenuator) in order to raise the UGF, and we eventually obtained ~125 kHz with a phase margin of ~23⁰ (see fig. 1).
We then checked the angular working point. Only minor corrections were needed, except for FF_V (+0.4 → +0.58). The new values have been applied and stored in the ACL.
Today, around 14h30 UTC, I have restarted on olserver117 (16 CPU) the process MdDQ which was running on olserver119 (4 CPU).
MdDQ starts each day at 6h30 the full reprocessing of the VIM plots of the previous day, so that they all start at 00h00 UTC (for the time plots) or are done at the max BNS range of the day (for the FFT plots). But, the number of scripts used in VIM has increased and this reprocessing was taking 100% of the 4 CPU and all the RAM of the olserver119 machine.
(see http://ganglia.virgo.infn.it/?r=month&cs=&ce=&m=load_one&c=New+Olservers&h=olserver119.virgo.infn.it)
and many scripts were killed during this daily reprocessing (as can be seen in /opt/w3/vim/util/reprocess_log.txt).
Thus, I have restarted MdDQ on olserver117, which has 16 CPU and much more memory. We will see if it solves the problem.
* PRELIMINARY *
Taking advantage of the UPS work at WEB (mains shutdown), we can compare the seismic, acoustic and magnetic noise levels of WEB (in purple all OFF, in blue standard conditions).
There is a huge difference in the triaxial magnetic fluxgate near the vacuum tank (ENV_WE_MAG_x), it seems that we have reached the sensor noise floor. To be further investigated.
Another interesting difference is the ALS microphone noise level (ENV_WE_ALS_MIC) for frequencies above 2 kHz.
HVAC was left always ON
* MORE DETAILED ANALYSIS WILL FOLLOW *
This morning around 12:40 UTC I realigned the MZI in SQZ box to recover some contrast, see picture.
I took the chance to check the optical power of the green beams:
- before MZI: 49 mW
- SQZ box output for FC: 32 mW
- after AOM first pass: 19 mW
- after AOM second pass: 12 mW
At 07:47:17 UTC, both ETM ring heaters were switched off.
At 07.59:44 UTC, Maria switched off the WE RH power supply.
========
At 12:38:09 UTC, WE RH was switched ON.
At 12:38:59 UTC, NE RH was switched ON.
The shift was dedicated to maintenance started at 6:00UTC, the main work of this morning was the replacement of WE UPS device; the switch off operations started at about 6:15UTC and they completed at around 8:00UTC; the UPS replacement completed at around 11:35UTC (#67178, #67178, #67182, #67181), now the equipments switch on restarted and it is ongoing; here a list of others activity reported to the control room:
- standard vacuum refill from 6:00UTC to 10:00UTC (VAC Team);
- cleaning of central building (Ciardelli with external firm: from 6:00UTC to 10:00UTC);
- DET: OMC Lock in single bounce configuration from 7:42UTC to 7:48UTC;
OMC scan in single bounce configuration from 7:52UTC to 8:13UTC;
- TCS: chiller check and refill (Ciardelli, 9:25UTC);
thermal camera reference 8:30UTC;
power checks 9:35UTC:
CH IN OUT
WI 0.305 0.042 0.499
NI 0.665 0.025 0.289
west end recovering ongoing.
SBE
SWEB control loops opened at 6:06UTC.
SUSP
WE suspension control loops opened at 6:10UTC.
This morning, Suzanne has switch off the PCal laser driver and Pd power supply before the WE UPS intervention:
LASER:
1435389674 01 Jul 2025 07:20:56 UTC (01 Jul 2025 09:20:56 CEST) PUMP OFF
1435389702 01 Jul 2025 07:21:24 UTC (01 Jul 2025 09:21:24 CEST) KEY OFF
1435389717 01 Jul 2025 07:21:39 UTC (01 Jul 2025 09:21:39 CEST) SWITCH OFF
POWER SUPPLY:
1435390544 01 Jul 2025 07:35:26 UTC (01 Jul 2025 09:35:26 CEST) POWER SUPPLY OUTPUT OFF
1435390579 01 Jul 2025 07:36:01 UTC (01 Jul 2025 09:36:01 CEST) POWER SUPPLY SWITCH OFF
WE PCal switched on by Suzanne around 13h UTC
LASER
1435409988 01 Jul 2025 12:59:30 UTC (01 Jul 2025 14:59:30 CEST) SWITCH ON
1435410007 01 Jul 2025 12:59:49 UTC (01 Jul 2025 14:59:49 CEST) KEY ON
1435410084 01 Jul 2025 13:01:06 UTC (01 Jul 2025 15:01:06 CEST) PUMP ON
1435410113 01 Jul 2025 13:01:35 UTC (01 Jul 2025 15:01:35 CEST) REMOTE ENABLED
POWER SUPPLY
1435410166 01 Jul 2025 13:02:28 UTC (01 Jul 2025 15:02:28 CEST) POWER SUPPLY ON
Then, at 13:08:00 UTC : enabled the laser from VPM and the control loops.
We then saw that there was extra-noise (see figure), and we realized that the LED flashing the IRIG-B in front of Tx_PD1 was enabled (red flag in the DMS, thanks Nicola for warning). SInce WE_Tx_PD1 is in loop, the noise was seen on Tx_PD2 and the Rx sphere instead.
At 16h28UTC, we disabled the LED: PCal noise level back to normal.
Yesterday evening injection had some troubles to keep the lock.
We suspected that it might come from the fact that the beam is quite on the edge of the FF QPD of the BPC (TY).
We moved the PR by 40 um in vertical to recover a healthiest working point on the quadrant (see attached plot).
The beam was still center on the ITM.
Here is the sequence of switch off actions performed on SWEB and WE device in order to prepare the shutdown :
2025-07-01-06h06m14-UTC info fabiog 'Bench loop:Open Angular Loop [-1:Disable,1:Enable=-1]' sent to SWEB_LC
2025-07-01-06h06m18-UTC info fabiog 'Bench:Open Position Loop [-1:Disable,1:Enable=-1]' sent to SWEB_SBE
2025-07-01-07h16m11-UTC info gouaty Command 'B8_QD1:close shutter' executed
2025-07-01-07h16m13-UTC info gouaty Command 'B8_QD2:close shutter' executed
2025-07-01-07h16m23-UTC info gouaty Command 'B8_QD1:disable vbias' executed
2025-07-01-07h16m24-UTC info gouaty Command 'B8_QD2:disable vbias' executed
2025-07-01-07h17m56-UTC info gouaty Power camera SWEB_B8_Cam1 off from mezzanine SWEB_MezzCAM
2025-07-01-07h17m57-UTC info gouaty Power camera SWEB_B8_Cam2 off from mezzanine SWEB_MezzCAM
2025-07-01-07h17m59-UTC info gouaty Power camera SWEB off from mezzanine SWEB_MezzCAM
2025-07-01-07h18m00-UTC info gouaty Power camera SWEB_ElectronicBox off from mezzanine SWEB_MezzCAM
2025-07-01-07h18m02-UTC info gouaty Close shutter of camera SWEB_B8_Cam1 from mezzanine SWEB_MezzCAM
2025-07-01-07h18m04-UTC info gouaty Close shutter of camera SWEB_B8_Cam2 from mezzanine SWEB_MezzCAM
2025-07-01-07h18m38-UTC info gouaty Close shutter of photodiode SWEB_B8_PD1 from mezzanine SWEB_MezzPD_B8
2025-07-01-07h18m40-UTC info gouaty Power down photodiode SWEB_B8_PD1 of mezzanine SWEB_MezzPD_B8
2025-07-01-07h19m23-UTC info gouaty 'PSD on/off [-1:Disable,1:Enable=-1]' sent to SWEB_PSD_sensing
2025-07-01-07h27m33-UTC info gouaty Disable output of PowerUnit13
2025-07-01-07h27m41-UTC info gouaty Disable output of PowerUnit12
2025-07-01-07h27m46-UTC info gouaty Disable output of PowerUnit19
PowerUnit13 switch off manually at 7h29 utc
PowerUnit20 switch off manually at 7h30 utc
PowerUnit19 switch off manually at 7h31 utc
SBE crate switched off at 7h37.
DAQ box 62 (SBE) switch off at 7h38 utc.
LVDT box (Nikhef voice coil control system) switch off at 7h38m30
DAQ box 43 (LC) switch off at 7h39 utc.
DAQ box 175 (NCal) switch off at 7h41
DAQ box 121 (TCS) switch off at 7h41m30
DAQ box 49 (PCAL) switch off at 7h42
DAQ box 50 (ALS) switch off at 7h44
SBE stepping motors and power electronic switch off at 7h45m30
The WRnode was powered ON last week, 2025-06-24 but after this , it was impossible to reconfigure the DBox_SN173 using the timing signal porvided by the WRNode
This morning , the WRnode was reseted, thank the reset buttom . After this operation the IRIGB was available at the DBox level and the DBox_SN173 was fully reconfigured .
The WR Timimg monitoring signals were restored in the TIMING_moni server .
Operations completed at 2025-07-01-06h38m06-UTC
ITF found in LOW_NOISE_3 and in Science Mode.
The ITF unlocked at 2:10 UTC and, after a period of instability of the Injection, it was possible to start the locking acquisition at around 2:22 UTC. After a series of attempts, the ITF was back in Science Mode from 3:25 UTC.
It unlocked again at 4:02 UTC, back in Science from 5:00 UTC.
Guard Tour (UTC)
20:55 - 21:38
23:18 - 23:56
1:18 - 1:57
3:16 - 3:54
At my arrival, locking acquisition was in progress.
ISC On-Call
18:30UTC, ITF unlocked five times at Acquire DRMI step or before (first when I disaligned the PR to align the CITF, see 1st plot), so I called D.Bersanetti to check the status
VPM
16:49UTC, "FmRds_II" process restarted after an unexpected loss of connection
Guard Tours (UTC):
17:50 - 18:30
20:55 -
Below the list of WE DSP glitches found without unlock of the ITF
ITF found locked at LN3. SCIENCE mode.
At 06:24 UTC I set ITF in ADJUSTING mode for a extraordinary nitrogen refill at WE. Unfortunately ITF unlocked during the truck transit.
Relocked at 08:13 UTC at LN3 after seferal attempts. SCIENCE mode set. AUTOSCIENCE engaged.
12:08 UTC, ITF unlocked (TBC). Relocking in progress...
Sub-system reportsISC
06:35 UTC, under request of Mantovani, we proceeded with a step of NI Etalon up to 20.24 in 8 hours.
As a side observation here you can find a trend of the DIFFp TY gain servo on Friday (Figure 1) and on Saturday (Figure 2).
