The switch box has been restored, the fault was simply a cable not well connected.
The behaviour of the MIR signals is identical to what it was before the intervention (fig.1 before, fig.2 after). We also restored the damper gain to its previous value.
The glitches observed on MIR PSDF in May (fig.3, 69083) are still there, although their shape is slightly different.
We turned off and removed the switch box between the MIR PSD and the respective power supply on the WI payload.
The MIR signals are now offline, and the damper's gain was set to 0 beforehand.
We will try to repair the switch box and reinstall it before the end of today's maintenance.
The switch box has been restored, the fault was simply a cable not well connected.
The behaviour of the MIR signals is identical to what it was before the intervention (fig.1 before, fig.2 after). We also restored the damper gain to its previous value.
The glitches observed on MIR PSDF in May (fig.3, 69083) are still there, although their shape is slightly different.
The planned activty of the shift was: TCS CARM_NULL_1F recovery ; it went on without major issues.
Activity concluded at 19:00 UTC; ITF left in locked arms IR.
The goal of this shift was to recover CARM_NULL_1F.
During the afternoon the ITF slowly continued to thermalize after the TCS actuators were enabled, causing the sidebands to rise in power at a steady rate that only started forming a plateau around 16:00 UTC (Fig.1).
This forced a continuous back and forth to adjust gains and phases, following this moving target, until the thermalization process stabilized.
Towards the end of our shift we managed to reach CARM_NULL_1F but could not stabilize this state, as a 11 Hz oscillation, slowly but steadily rising, eventually breakes to lock. The main culprit seems to be SRCL gain being too high, with the oscillation being wide enough to overwhelm the UGF servos, preventing them to adjust the gains. This oscillation is witness by a vast number of longitudinal and angular loops, making diagnosis more difficult.
Overall, the demodulation phases are adequate in 3F and good enough in 1F to allow the transition, but we could not complete their fine tuning before the end of the shift.
A fourth event occurred the 2026-09-13-11h28m55-UTC and was promplty recovered .
This morning, we first checked that the CO2 actuator powers were still consistent with the values set last week, based on the OSCAR simulations and summarized in the table below:
| OSCAR simulation Starting point | CH [W] | INNER DAS [W] | OUTER DAS [W] | |
| W | on the ITF | 0.022 | 0 | 0.4 |
| on the pickoff | 0.135 | 0 | 0.065 | |
| N | on the ITF | 0.022 | 0.15 | 0.6 |
| on the pickoff | 0.135 | 0.024 | 0.098 |
Only a small adjustment of the NI CH power was needed, probably due to the recent interventions on the cooling system. At the end of the check, all the CO2 flip mirrors were closed again.
A first attempt to reach CARM NULL was performed around 08:10 UTC, but it was not successful. No clear evidence of PA was observed on the ITM thermal cameras.
We therefore decided to return to CITF and proceed with the switching ON of the CO2 actuators. CITF 1F was reached at around 08:44 UTC, with the following initial sideband levels:
112 MHz: 0.027 mW
12 MHz: 0.051 mW
At 08:45 UTC, the DAS flip mirrors were opened, with the powers reported in the table above. The evolution of the sideband levels was then monitored for almost two hours. A general worsening was expected with the DAS actuators ON alone.
After about one hour, at 09:55 UTC, the 56 MHz and 6 MHz sideband levels had decreased by approximately 30% and 20%, respectively. We agreed to manually unlock the interferometer at 10:10 UTC.
The ITF returned to CITF 1F at 10:20 UTC, with the following sideband levels:
112 MHz: 0.0195 mW (-27%)
12 MHz: 0.0355 mW (-30%)
The levels were then quite stable. The trend during this first period is shown in Figure 1.
At 10:34 UTC, the CH flip mirrors were opened, again with the powers reported in the table above. The sideband levels increased almost immediately. Already after about 20 minutes, at 10:55 UTC, they had nearly recovered their initial values.
After the lunch break, some adjustment of the PR angular loop gain was needed to allow the ITF to lock again in CITF 1F, which was achieved at around 12:05 UTC. The sideband levels then reached:
112 MHz: 0.0305 mW (+12%)
12 MHz: 0.0544 mW (+7%)
The trend during this second period is shown in Figure 2.
Just before the Daily Meeting, the first attempts to reach CARM NULL started.
The activity continued with the afternoon shift.
ITF found in COMMISSIONING Mode and LOCKED_ARMS_IR State,
All times are UTC.
07:06 Start of commissioning activity - Turn on DAS/CH in CITF (Lumaca, Corubolo, Nardecchia, Bersanetti, Spinicelli).
The shift was dedicated to TCS activity with centra interferometer lock. Activity still ongoing.
ITF left in COMMISSIONING Mode and LCOKED_ARMS_BEAT_DRMI_1F State.
The only remaining ISC related explanation that I can think of to explain the large decrease in sidebands are large offset in PRCL or MICH.
Figure 1 shows an example of a lock acquisition on Friday. Around 15:24 UTC there is a transition where the B2 8MHz I signal becomes 3 times smaller, other B2 signals have also a step, and the SSFS line height in B2 DC decreases by a factor 4. But it has no incidence on the sideband recycling gain. This step behaves like a large improvement in PRCL offset, and thas not impact on the sideband gain, which would mean that a PRCL offset doesn't play a role in the sideband gain decrease.
On the other hand at 15:20:30 UTC there is another large transition. Where the sideband gains decreases, especially the 6MHz. At the same time the misbalance of the 56MHz sideband on B1p increases by a factor 3, and the 6Mhz power on B1p also increases by almost a factor 2. This misbalance then continues to increase over time, the B1p 6MHz power increases over time, and the sideband gain decreases. This could be explained by a very large offset on MICH, that then continues to increase over time.
Figure 2, 3 and 4 shows phase camera images on B4 and B1p during the CARM null lock. The sidebands on B4 progressively move away from the position of the carrier on B4. And the shape of the 6MHz sideband, and 56MHz LSB becomes more and more like a doughnut, which would be a sign of a mismatch between the north and west short Michelson, which could mean one of the input mirrors absorbs much more than the other.
After adjusting the B1 PD3 offset the DET_MAIN checks of the slow shutter were successful and I could relock the arms. The offset adjustment will probably need to be undone in a few hours as the temperature of the SDB2 bench is slowly coming back up once it is back to be supplied in power.
I ran an OSCAR simulation considering an input power of 12 W and an absorption of 0.3 ppm, as specified by LMA for the new mirrors. With these values, the absorbed power is about 20 mW.
In the absence of thermal effects (i.e. in the ideal case), the gains are about 74 and 51 for the 6 MHz and 56 MHz sidebands, respectively, assuming 1500 ppm losses in the recycling cavities.
No cold maps of the mirrors were included, except for the POP cold maps. I assumed PR and SR RoCs of 1430 m, RoCs of 1427.8 m and 1428.8 m for the new input mirrors, and RoCs of 1670 m for the end mirrors, corresponding to the expected value with the RHs on.
Including the OPL variation induced by 20 mW of absorbed power, shown in the attached figure, the 6 MHz and 56 MHz sideband gains decrease to about 29 and 22, respectively. The gains are calculated considering all modes, not only the TEM00.
To obtain a 6 MHz sideband gain of about 10 in the simulation, the assumed absorption needs to be increased by approximately a factor of 3, from 0.3 ppm to 0.9 ppm. Equivalently, keeping the absorption at 0.3 ppm, the same thermal load would correspond to an input power of about 36 W.
For the third time, the SDB2_DBOX_LeftUp became unreachable on the Tolm network at 2026-09-12-21h02m15-UTC
Operations performed between 2026-09-13-05h56m37-UTC and 026-09-13-06h16m38-UTC
The attached plot shows the SDB2 DaqBoxes timing error signals, their +/-12V power supplies monitoriing channels and each DBOX external temperature sensors (the one of RightDown doesn't work)
Yesterday morning I found all the ALS photodiodes at NE giving strange values (see fig. 1). Looking back in time, it appears that right after the opening of most of the suspension guardians on Thursday night, all the ALS photodiodes (CEB, NE, WE) went to 0 several times as a consequence of the frequent unlocks. However, at NE those values stucked at 0 at 22.20UTC and almost one hour later they went to negative values (fig. 2). Also the SOA stopped to work.
Investigating the electronics of the power supply, we find out that apparently there was a loose connector on the +24 DC distribution box that started to became a problem only on Thursday night for no apparent reason.
Repairing the connector solved the problem. The NE ALS signals are stable since the restart of the photodiodes and SOA.
Yesterday afternoon, I did a first check of the NI mirror during the short CARM NULL locks of yesterday, no PAs has been observed so far, but the focus point of the camera has to be better set next days.
Moreover, profiting of some down period during the commissioning shift at the end of the afternoon, I installed a new thermocamera at the WI, at the small ZnSe viewport access.
As usual, the camera stream is accessible only from devices connected to the EGO WiFi network, via the address : http://172.31.10.153:8080/. Data will be acquired next week. Both camera should be accessible at the same time.
In order to check if ISC actions could help in the survival of the CARM null 1f step we have checked and tuned:
- SSFS phase and gain (tuned)
- PR tx and ty phase
- B2 8MHz (input beam alignment)
- SR alignment behavior
- DiffP set servo
None of these actions helped in reducing the drop of the sidebands
We tried also a lock acquisition with the SR misaligned -2urad in ty. No effect.
The only remaining ISC related explanation that I can think of to explain the large decrease in sidebands are large offset in PRCL or MICH.
Figure 1 shows an example of a lock acquisition on Friday. Around 15:24 UTC there is a transition where the B2 8MHz I signal becomes 3 times smaller, other B2 signals have also a step, and the SSFS line height in B2 DC decreases by a factor 4. But it has no incidence on the sideband recycling gain. This step behaves like a large improvement in PRCL offset, and thas not impact on the sideband gain, which would mean that a PRCL offset doesn't play a role in the sideband gain decrease.
On the other hand at 15:20:30 UTC there is another large transition. Where the sideband gains decreases, especially the 6MHz. At the same time the misbalance of the 56MHz sideband on B1p increases by a factor 3, and the 6Mhz power on B1p also increases by almost a factor 2. This misbalance then continues to increase over time, the B1p 6MHz power increases over time, and the sideband gain decreases. This could be explained by a very large offset on MICH, that then continues to increase over time.
Figure 2, 3 and 4 shows phase camera images on B4 and B1p during the CARM null lock. The sidebands on B4 progressively move away from the position of the carrier on B4. And the shape of the 6MHz sideband, and 56MHz LSB becomes more and more like a doughnut, which would be a sign of a mismatch between the north and west short Michelson, which could mean one of the input mirrors absorbs much more than the other.
In view of the next commissioning activities, we set up a very basic (and temporary) check on the power reflected by the IMC (INJ_IMC_REFL_DC); this is done only in the IMC_RESTORED state of the INJ_MAIN node and, if failed, it will prevent moving forward to the lock acquisition if such power is out of a specific range (+- 50 mW around the current average, changes may still happen); this is to avoid bad lock attempts after unlocks following manual or automated changes of the input power, especially for lower power that may make the lock of the IMC itself less reliable.
A forced failure of the check has been tested, and the new cyclic decorator is now in operation.
06:00 UTC ITF found with the following loops open:
ITF was put in DOWN with ITF_LOCK and INJ_MAIN nodes in pause.
I was able to close some of them but SIB1 and WI had too high a correction to hold. Manuel started working on the issue while I called Valerio (work in progress), since we noticed that many stages of different suspensions were close to saturation and needed to be moved with motors.
SBEs closed.
08:15 UTC I changed the TCS Aux cooling chiller setpoint from 17 to 16.1. Heating loop setpoints: WI 17.4, NI 17.2. CO2 CH powers are back to before adding glycol.
10:00 UTC ALS NE malfunction assessment: L1 power supply not working (Cavalieri, Spinicelli)
12:00 UTC ITF in LOCKED_ARMS_IR
Figure 1. The 6MHz sideband gain drops from 60 to around 10, and the decrease is still on-going when the unlock happens. During O4 in CARM NULL the sideband gain was around 23. Maintaing the CARM NULL lock in this case for longer than 5 minutes may be a challenge without thermal compensation, or at least requires UGF servos and MICH/PRCL/SRCL diagonalization that can react on a time scale of one minute.
The automated lock loss analysis points the finger at DIFFp low gain as the reason for the unlock. Figure 2 and 3 show that in two examples where the interferometer stayed in CARM NULL for a few minutes the DIFFp gain servo has been struggling, with the gain adjustment oscillating by a factor 2 or 3 on a time scale of minute, with increasingly large oscillations.
The activity of the shift was the CARM_NULL_1F recovery.
In agreement with Mattia I kept trying to lock up to 20:20 UTC when I park the ITF in locked arms.
Automation
At 20:13 UTC the Automation nodes of DRM_LOCK, ITF_LOCK and SUSP_PR got stuck: to restore the situation I restarted the nodes and performed some INIT cicles.
DAQ
SuspFB restarted at 15:42 UTC to recover data after an event of missing data.
The goal of this shift was to recover CARM_NULL_1F.
During the shift we suffered from issues related to a few passages in the lock acquisition:
It should be noticed that CARM_NULL_3F remains a highly unstable state that only lasts long enough to go to 1F. In our case it worked 3 times.
The initial attempts to engage CARM_NULL_1F failed for what seemed to be a wrong phase for B4_56MHz. We attempted to rotate it by 45 degrees and we were lucky enough to pick the right way, yielding a successful transition for the next attempt. We then proceeded to fine tune both B4_56MHz and B1p_56MHz phases to minimize the MICH_PRCL_COUPLING and the DPHI channel respectively.
The first proper lock in CARM_NULL_1F (the first actual one was accdentally killed by human error) lasted for about 4 and a half minute (Fig.1). The sidebands power in the PRC drops by about 60% in this time.
We leave the ITF chasing CARM_NULL_1F to accumulate potentially useful data for TCS and instruct the operator to park it in locked arms before leaving at 11pm.
Figure 1. The 6MHz sideband gain drops from 60 to around 10, and the decrease is still on-going when the unlock happens. During O4 in CARM NULL the sideband gain was around 23. Maintaing the CARM NULL lock in this case for longer than 5 minutes may be a challenge without thermal compensation, or at least requires UGF servos and MICH/PRCL/SRCL diagonalization that can react on a time scale of one minute.
The automated lock loss analysis points the finger at DIFFp low gain as the reason for the unlock. Figure 2 and 3 show that in two examples where the interferometer stayed in CARM NULL for a few minutes the DIFFp gain servo has been struggling, with the gain adjustment oscillating by a factor 2 or 3 on a time scale of minute, with increasingly large oscillations.
I ran an OSCAR simulation considering an input power of 12 W and an absorption of 0.3 ppm, as specified by LMA for the new mirrors. With these values, the absorbed power is about 20 mW.
In the absence of thermal effects (i.e. in the ideal case), the gains are about 74 and 51 for the 6 MHz and 56 MHz sidebands, respectively, assuming 1500 ppm losses in the recycling cavities.
No cold maps of the mirrors were included, except for the POP cold maps. I assumed PR and SR RoCs of 1430 m, RoCs of 1427.8 m and 1428.8 m for the new input mirrors, and RoCs of 1670 m for the end mirrors, corresponding to the expected value with the RHs on.
Including the OPL variation induced by 20 mW of absorbed power, shown in the attached figure, the 6 MHz and 56 MHz sideband gains decrease to about 29 and 22, respectively. The gains are calculated considering all modes, not only the TEM00.
To obtain a 6 MHz sideband gain of about 10 in the simulation, the assumed absorption needs to be increased by approximately a factor of 3, from 0.3 ppm to 0.9 ppm. Equivalently, keeping the absorption at 0.3 ppm, the same thermal load would correspond to an input power of about 36 W.
This morning I found the cavities locked on the infrared; the ITF lock recovery went on all the morning without any major problem, the ITF was locked up to CARM_NULL_3F, the work will go on in the afternoon...
Sub-system reportsTCS
13:35 UTC auxiliary cooling system chiller setpoint changed from 17.8 to 17. Heating control loops setpoint changed accordingly (Cavalieri, Zaza)
Today around 13:38 UTC the setpoints of the etalon loops have been changed to:
Yesterday afternoon we made some additional test to confirm that this additional noise was coming from the new LNFS.
We used one of the two free output of the LNFS used now for the 22MHz to generate a twin 56MHz signal to feed *only* the demodulation box (not the EOM). In fig. 1 we can observe that the noise disappears in the LNFS dfreq channel, but not in the EOM one, confirming that there is an issue with this device.
For the time being, the only one possibility left for the generation of the 6/8/56MHz is to use the LNFS used up to Monday morning, which appeared to be more noisy than usual , but good enough to work on the interferometer.
We replaced the old LNFS at 14.45UTC.
The afternoon and the evening were spent working on Lock recovery up to STEP 3 #69724 (Bersanetti, Spincelli)
Commissioning activity ended at 19:30 UTC
21:00 UTC ITF left in LOCKED_ARMS_IR state and COMMISSIONING mode
Guard Tour
18.35 UTC
After the replacement of the LNFSs to the configuration that was in use last Monday afternoon, we reverted the automation configuration files to the svn snapshot in use at the time , in order to have a usable starting point.
We then proceeded to examine all steps of the lock acquisition from DRMI_1F onwards. We did notice some changes also at that stage, in terms of gains and demodulation phases. As usual, the 3F step was the most impacted.
Then we moved to the CARM/DARM handoffs to the IR, and the following STEP1/2/3 states of the CARM offset reduction, which had to be tuned more heavily.
We consider the recovery relatively reliable up to STEP3, with the following observations:
