As stated in multiple occasions, there is no Metatron glitch unlocking the interferometer. The unlock was at the PSTAB level (see Figure).

There is, after an unlock, the bug by which sometimes the request to go back to LOW_NOISE_3 is overwritten by the last request done by CALI, even if there shouldn't be one. Yesterday ITF_LOCK was restarted (for another reason) and the bug persists. While a proper fix is still missing we could try to stop and restart also CALI.

Today, I found ITF in SCIENCE. At 21:10 UTC, the ITF unlocked. For some unknown reasons, the ITF started to experience problems related to INJ. While trying to perform Fmoderr, the node got stuck in a loop. I temporarily put down the INJ_MAIN node then restarted the lock, and for some reason, this procedure worked, allowing the ITF to reach CARM_NULL_1F.

However, we then encountered the same issue as in previous days. Due to a glitch, Metatron requested LOCKED_PRWI while at CARM_NULL_1F, which caused the system to unlock. While attempting to relock, INJ_MAIN got stuck again in the Fmoderr loop.

To resolve this, I modified the ITF_LOCK ini file to skip the Fmoderr check by changing the line fmoderr_skip = false to fmoderr_skip = true. Thanks to this change, I was able to restart the lock successfully.

At 23:31 UTC, the ITF was back in SCIENCE.

At 00:04 UTC, the ITF unlocked.

At 00:53 UTC, the ITF was back in SCIENCE.

At 02:36 UTC, the ITF unlocked again; the WARM_LOCK process crashed, and I had to restart it. I reverted the change on the ITF_LOCK ini file from fmoderr_skip = true back to fmoderr_skip = false. I also manually realigned the CITF. After an unlock at LOW_NOISE_2, 04:52 UTC the ITF finally reached SCIENCE.

ITF found relocking. PREPARE_SCIENCE mode.
At 17:40 UTC ITF unlocked due to a failure of ISYS system. Laser amplifier switched OFF due to a failure of Neovan Chiller (Low level of water). I contacted ISYS oncall.

TROUBLESHOOTING mode set from 17:50 to 18:56 UTC.

With Camilla we checked for leaks and recovered the correct level of chiller. ISYS system recovered in the standard state.
After a little realignment of North cavity we restarted with the lock acquisition.
ITF back in SCIENCE mode at 20:50 UTC. Autoscience enabled.

Guard tour (UTC):
17:50 - 18:33
20:55 - 

DET
during the lock acquisition it was needed to rearm Vbias and open shutter of B1p_QD1 that was disabled. Twice!

ISYS
(11-07-2025 18:30 - 11-07-2025 19:00) On site
Status: Ended
Description: Neovan chiller failure.

The following report has been submitted to the On-call interface.

On-call events -> Injection System

Title: Chiller of Neovan amplifier head off

Author(s): derossi

Details:

The Neovan amplifier went in protection mode because the chiller cooling its head switched off due to the low level of the water. This is strange since we refilled it last Tuesday (along with the change of the filter). For this reason with Nicola Menzione we carefully checked for some leaks on the laser bench and under the floor, but we couldn't find any. We thus refilled the water and restarted the system.

* Note that any files attached to this report are available in the On-call interface.

An attempt to produce another polynomial reconstruction of the sensitivity has been done. The first problem to face is the presence of many glitches, reducing the length of the available good segments of data. We used a trick which seems acceptable: from a long period of continuous lock in the same condition, a list of segments has been selected, putting  a threshold on BNS range and considering only the segments long enough. On those data, an amplitude spectral density has been computed, making the average of the averages.

This operation has been done for the data acquired on Jul 4, starting from 16:40 UTC, for 70000 seconds. The BNS range was high thanks to the offset on SR_TX. Eleven segments with BNS range > 55.8 Mpc has been selected for a total time of 2156 s, obtaining an averaged range of 56.5 Mpc.

The floor of the spectrum has been reconstructed by a quadratic sum of simple contributions

• 1/f^4      3.2e-23 @ 35 Hz   (control noise)
• 1/f^3      0.29e-23 @ 60 Hz (control noise) a small adjustment of the low frequency part, dominated by the previous term
• 1/f^2.5   30e-23 @ 10 Hz (pendulum thermal noise)
• 1/f^0.5   0.55e-23 @ 100 Hz (coating thermal noise)
• 1/CavPOLE   0.55e-23 @ 0 Hz (readout noise)
• 1/(CavPOLE*f^0.5)    0.6e-23 @ 100 Hz (mystery noise);   the amplitude regards the 1/f^0.5 part, before the cavity pole.

The list of contribution is a bit arbitrary and doesn't want to be a real noise budget: the only check that it could be a good model comes from the quality of the final reconstruction, but other models could be equally good or better.

The final reconstruction is the output of a semi-automatic iterative process, which returns the 6 amplitudes shown in the list. It is a search of a minimum error, but not on 6 free parameters: the two regarding the thermal noise are fixed, coincident with an expected value. A check on those two values has been done, observing that the quality of the fit starts to degradate if too different vaues are choosen. A full automatic search was unsatisfactory, because too dipendent on the starting values.

A note on the mystery noise slope: putting it as a parameter of the search, the output was nomally closer to -0.5 than -0.66. Using it as a fixed number, the results were normally more accurate with -0.5, than -0.66. A fixed number -0.5 has been considered the best choice: using slopes as parameters gives not well converging results and seems not a good way to proceed, if the aim of the study is the evaluation of contribution amplitude in different conditions. As a personal opinion, -0.5 slope is preferable also in case of equivalent fitting accuracy, because it has more probability of being associated to a fundamental process.

The optimal values of the parametric amplitudes have been obtained after a double iteration. The first has been applied to all the dots in the interval 15-2000 Hz. Using the first result, several dots has been excluded: all the ones too far from the fit. The aim of this selection was to exclude peaks, bumps and lines from the curve and keep only the floor. A second search has been done on the selected data.

Fig 1 shows the reconstruction for 56.5 Mpc curve.

Fig 2 shows shows the accordance between the selected data and the fit. The yellow dots are residual values for all the data, re-including the dots excluded after the first iteration.

Fig 3 shows a similar analysis performed on data acquired with SR_TX aligned, starting from Jul 3 20:05, 45000 s. THe threshold on BNS has been put at 53.5 Mpc. The averaged range is 54.2 Mpc. The new amplitude of the mystery noise is 0.68e-23; the one with the adjustment of SR_TX was 0.60e-23. The output was slightly different also for other two components: the 1/f^3 and the readout noise. For both, the amplitude is about 6 % lower than the previous case. At high frequency the variation could be explained by an increase of the optical gain; the variation at low frequency could be not significant because regards a small component of uncertain origin.

Fig 4: the same analysis has been applied to data in LN2, SR_TY aligned and mystery noise dominant in the data (Jul 3 15:45). Also for those data, the slope -0.5 seems to me preferable, even if the quality of the fit is less good (see also fig 5).

Fig 6: the recontruction done for the curve at 56.5 Mpc is used to show how a curve without mystery noise would be. The BNS range would reach 69 Mpc and there would be still a lot of noise to remove, before reaching the supposed limit of fundamental noise for this configuration, at 89 Mpc.

ITF found relocking at LOCKED_ARMS_BEAT_DRMI_1F in PREPARE_SCIENCE (Autoscience ON). Achieved LN3 at 05:36UTC.
Almost immediately unlocked at 05:43UTC (pdf 1) due to BS glitch with ITF_LOCK request=LOCKED_PRWI, as has been the case for the 3 most recent unlocks (fig.1).
Automation stuck in LOCKING_ARMS_BEAT_DRMI_1F, fixed by performing cross-alignment in ACQUIRE_DRMI with Operator-in-trainining Zaza (SR_TY was misaligned by 12 urad).
Lock recovered at 07:14UTC, back in SCIENCE.
ITF set to DOWN in TROUBLESHOOTING mode at 09:50UTC to allow for CEB interventions:

• 09:55-10:51UTC swap of DSP card in BS rack, substitution of fan filters, resync of Timing_dsp by Boschi and Piendibene
• 10:00-10:20UTC inventory work in cabling lab by Dattilo.

Lock recovery after BS intervention (UTC):

• 10:53 Closed SPRB_SBE position loop from VPM.
• 11:02 Realigned NE cavity for ARMS_LOCK on IR.
• 11:15 usual cross-alignment in ACQUIRE_DRMI. SUS_PR node stuck while MISALIGNING. Only fixed by restarting ITF_LOCK at 11:40UTC.
• 11:48 Unlock due to RFC unlock during second attempt at DRMI alignment. INJ on-call (De Rossi) notified. Automatically relocked.
• 12:12 Succesful cross-alignment in ACQUIRE_DRMI by Zaza, Bersanetti.
• 12:56 Relocked in LN3, back in SCIENCE with Autoscience ON.

ITF unlocked almost immediately again at 13:04UTC due to WE glitch.

ISC
07:24UTC Violin modes excited for a few minutes, see DMS flag in fig. 2

SQZ
10:22UTC Intervention on VAC system by Vardaro

ITF found in Science mode.

The ITF unlocked twice (22:10 UTC and 23:57 UTC); every times it relocked at the first attempt.

It unlocked again at 4:48 UTC, relock in progress.

Guard tours (time in UTC)

22:00-22:40; 0:00-0:40; 2:30-3:10

Today, I found the ITF in SCIENCE mode. At 15:10 UTC, the ITF status was in COMMISSIONING. Here is the list of activities:

• Switch on auxiliary energy system at the TB for 15 minutes from approximately 15:15 to 15:30 UTC (M. D'Andrea)
• Test of the new version script for ENV injections (M. Tringali, D. Bersanetti)
• SAT BS board test (V. Boschi)

Here is the list of activities I performed:

• 17:56–18:16 UTC: ASC injections
• 18:18–18:37 UTC: CALIBRATED_DF_SENSITIVITY
• 18:37–18:55 UTC: UTF DF_PCAL
• 18:57–19:43 UTC: ENV noise injections

19:47 UTC: ITF back in SCIENCE mode

Thanks to Diego's help, today we tested the code for performing acoustic injections at the terminal buildings via the ENV_MAIN Metatron node. Figures 1–2 show the colored noise sent to the DAC channels at CEB, NEB, and WEB (amplitude = 0.04, duration = 120 s, freq. range =  8-2000 Hz).
Unlike at CEB, the loudspeakers at the terminal buildings are not yet connected to the smart PDUs. However, the on/off switching process of the dedicated sockets was triggered by the code and worked as expected.

Below are the PDUs and the socket used:

• WEB: PDU_WE_ALS_NNC, socket 3
• NEB:  PDU_NE_ALS_NNC, socket 3

Powering the loudspeakers via smart PDUs at the terminal buildings will be implemented in the coming weeks.

ITF found locked at LN3 in SCIENCE mode with a BNS Range ~56 Mpc.
06:10 UTC - Under request of Mantovani i set ITF mode in ADJUSTING for a minute to perform a step of NI Etalon from 20.34 to 20.4 in 36000 sec.
At 06:52 ITF unlocked due to the unlock of IMC. The LC of SIB1 opened by the guardians. I closed it properly but the RFC didn't relock automatically.
TROUBLESHOOTING mode set from 07:25 to 08:00 UTC.
I called ISYS OnCall expert. Derossi recovered the correct bench position.

After the unlock of 07:00 UTC the DET_MAIN metatron node got stuck in "unknown" state. Properly recovered via INIT.
At 07:00 UTC I re-Load ITF_LOCK. (Bersanetti request).

Relocked at first attempt. SCIENCE mode set at 08:47 UTC. AUTOSCIENCE enabled.
Unfortunately ITF unlocked again. Ruggi set again a good configuration of Reallocation.
Autoelocked at first attempt. SCIENCE mode set at 09:47 UTC.
There is still a communication problem between DSP (LVDT board for the horizontal loop control) and FO of BS. I contacted Suspension expert and informed him of this issue. An intervention in CEB (BS rack) has to be planned.

SUSP
05:19 UTC - LargeCoil_WEB process crashed / restarted

ISYS
(10-07-2025 07:30 - 10-07-2025 08:00) On site
Status: Ended
Description: RFC didn't relock automatically after the INJ unlock.
Derossi checked on it and recovered the bench position.

I ve changed the upper set of the SRCL set clip from 90 to 110 since during last nigth it was saturated at 90 (it will be changed also in the lsc config file of Acl to be robust against Acl restarts)

This morning at 5:12 UTC I have put ASC_SR_TX_SET to 0.022 with a 2 hours long ramp. We have over 12h of steady lock before that, so this change should give us another check if that offset remains a good way of gaining a few Mpc.

At 6:52 UTC the interferometer unlocked, so the ramp did not get the time to arrive to the end. However looking at the trend over the preceeding ~8 hours, clearly the BNS range improved by ~2 Mpc. Les clearly the dark fringe power reduced by a few percent, the DARM optical gain may have decreased slightly. The SR change in alignment is clearly visible on the B1p quadrant DC channels, especially on QD2 V and QD1 H. 

ITF in Science mode for all the shift.

Guard tours (time in UTC)

23:00-23:40; 1:00-1:40; 3:17-4:00

Today, I found the ITF in SCIENCE.  It remained locked for the rest of the shift.

Guard tours (UTC):

17:17-17:47 

Since yesterday during the maintenance a new problem is present: the ITF unlocks quite often and most of the times it happens soon after a fast glitch on BS top stage data. Togethere with the time coincidence, the evidence of correlation is given by the fact that the loss of ITF control happens because of an increased longitudinal correction on BS payload, and only on BS.

The glitches on BS top stage are much more frequent than the unlocks and the majority of them have apparently no effect on the locking: there is something not clear in the unlock mechanism.

The data affected by the glitches are only the top stage position signals, reconstructed from the three horizontal LVDTs; in the raw LVDTs, the glitch is not visible. The value of the reconstructed signals during the glitch are the ones generated by the constant set point added in the CoilDriver board before generating the reconstructions; this is compatible with zero values received by the CD board from the LVDT board.

Trying to mitigate the effect of the glitches, the BS reallocation to IP has been disabled. After that, the lock is surviving more than before, but we need to consider also the fact that the rate of glitches is decreasing.

ITF found relocking at CARM_NULL_1F and in PREPARE_SCIENCE with Autoscience enabled.
After the unlock of 05:35 UTC the DET_MAIN metatron node got stuck in "unknown" state. Properly recovered via INIT.
No problems with the lock acquisition. Investigation in progress, by ISC experts, on recent unlocks events. Probably a glitch on BS IP.
Relocking in progress...

DET
10:23 UTC - B1p_QD2 stuck during the lock acquisition. Properly closed via VPM.

This morning, during an unlock phase of the interferometer, we stopped the main turbo pump of the PR tower, since it was showing signs of malfunction (power instabilities).

In its place we have activated the spare pump, now operating normally, and during the next two tuesday breaks we will replace the main pump with a new unit .

ITF found during locking acquisition.
During the shift the ITF unlocked multiple times from LOW_NOISE_3, with the longest lock lasting 40 minutes. The others round up to 15 minutes.

Here the timetable of the events:
21:38 UTC - Science Mode
21:53 UTC - Unlock
22:48 UTC - Science Mode
23:29 UTC - Unlock
00:19 UTC - Science Mode
00:36 UTC - Unlock
1:48 UTC - Science Mode
1:55 UTC - Unlock
2:50 UTC - Science Mode
3:02 UTC - Unlock
4:13 UTC - Science Mode
4:29 UTC - Unlock

Relock in progress.

ITF found in relocking phase, PREPARE_SCIENCE with Autoscience enabled.

At around 13:07 UTC, while the ITF was in CARM_NULL_1F, EIB started to oscillate; I was able to reclose the loop, then Camilla and Mathieu (present in control room) recovered the standard state for the IMC.

At 14:08 UTC ITF in Commissioning mode for  Iterative tuning of subtraction filter gains; activity concluded at 18:43, ITF set in Science mode.

The ITF unlocked and relocked other two times; the lock acquisition is working properly while the lock reliability seems to have some problem. To be checked.

ISC

At 18:35 UTC under request of Maddalena I changed the etalon loop: 2025-07-08 18h35m53 UTC    berni 'WI:WI Set point [WI_RH_SET=20.8,rampTime=36000]' sent to LSC_Etalon_Acl

Guard tours (time in UTC)

17:52-18:30

The goal of the shift was to test a new algorithm that can iteratively optimize the gain of a subtraction filter (and later estimate the filter itself) using only science data, so without using noise injections.

We started the shift by setting the gain of the MICH subtraction filter to 0.1 at 14:20:29 (the standard one is 1.02). We ran the algorithm we prepared in MATLAB which found a new gain of 0.1214 (which we set at 14:38:30). However the increments with which it updated the parameter were way too low. For a few hours we tried to find the solution why the increments are so small but we haven't been able to find the issue and we can't reproduce this in simulation, where it works fine. We gathered some additional data at different subtraction filter gains to test this after the shfit to see if we can find the solution in our algorithm. We set these gains at:

16:54:30 - 0.5

17:13:25 - 0.8

With this we have enough data to hopefully find the solution. In the meantime during the shift Diego also performed some work and we unlocked twice (more about this in separate post). 

This is a comparison of data in LN3, for the analysis of the new DIFFpTX control.

Fig 1: this morning, in the latest lock before the change of DIFFpTX control, the microseism was a bit higer in the critical region around 0.75 Hz. The following comparisons should take into account this difference.

Fig 2: The locking correction of West Arm sees this difference: the amplitude of the payload resonances are lower by a bit less than a factor of 2.

Fig 3: Pitch on COMMp angular dof is lower too; here the difference is a bit larger.

Fig 4: On DIFFpTX the difference is more evident: up to a factor of 3. This should be due to the improvement of the control filter.

Fig 5: One should expect an improvement of dark fringe stability, but not so much.

Fig 6: In the morning data, the 2D graph DIFFpTX Vs B1p seems to say that B1p fluctuation is entirely due to DIFFp TX. The same analysis applied to the afternoon data does not agree with the previous one: it seems that the coupling has changed a lot.

Fig 7: There is a simple explanation: ASC_DIFFp_TX_CORR is the signal after the loop gain, but the gain has changed by a factor of 2. I had forgot about the automatic adjustment of the gain. In principle, this adjustment should guarantee a sort of constant calibration for ASC_DIFFp_TX_CORR: for this reason I prefer to use the signal after the gain to make quantitative comparisons. But in this case the assumption is clearly wrong.

Fig 8: The 2D anaysis done with DIFFp_TX before the gain. Here the calibration is more realistically the same. The large improvement of DIFFp_TX and B1p_DC is not only due to the change of control filter, but also to the increase of the gain.

We have learned that the gain was lower than the optimal one for a good control accuracy. It is not clear why with different controls the adjustment of the gain works in a different way. By the way, looking at older data, it seems that in the recent data the gain was lower than usual.

The last point: the new filter has a roll-off not well refined and the noise at 20-40 Hz could be too high. A new version has been downloaded. No difference is espected in the control band, but we will check in the next lock acquisition that everything works fine.

This morning we finished the installation of the PDU switches in the remaining towers (Injection Bench, North and West Input, and Beam Splitter). For each PDU, we tested the proper functioning of all stepper motors.

Operation Start Time: 10:45 (local time)
NI: 10:45 - 11:06
BS: 11:07 - 11:19
WI: 11:20 - 11:30
IB: 11:32 - 11:43
Operation End Time: 11:43 (local time)

From 11:43 to 11:53 we took the chance to remove the clogged air filters from IB, WI, BS, NI and SR. They will be cleaned and re-installed on the next maintenance.

Trying to explain the change in sensitivity during SR TX misalignment with a 1/f^{2/3} noise change corresponds to a 20% change in noise level.

Figure 1 shows the simplified noise budget during the night before the SR TX offset

Figure 2 shows the simplified noise budget during the SR TX misalignment, near the peak of sensitivity

Figure 3 compare the sensitivity between the two times.

The simplified noise budget are not perfect fits for the data, so there is room enough for errors, and it could be some other kind of noise that is changing. As noted before the frequency noise gets worse, when the BNS range is improving.

/users/mwas/detchar/toySensitivity_20250703

Yesterday morning, when the seismic condition started to be quite haevy, we noticed a very unstable dark fringe (fig 1). The fluctuation was entirely due to DIFFp_TX (fig 2), the spectrum of which was dominated by a large peak at 0.75 Hz (fig 3). The sensitivity of pitch to longitudinal motion is a well known problem: the peak at 0.75 Hz is the second resonance of the payload and its coupling to the angular motion is not strange, but yesterday it looked a bit too strong and it would be better to understand is everything is working properly at the level of suspension control.

So far, we didn't find anything broken, but we found that DIFFp_TX control filter has some margin of improvement concerning the gain at 0.75 Hz. We developed and implemented an improved control. We tested it this morning, during the first lock acquisitions after the maintenance. We have data in similar seismic condition (fig 4), confirming what expected from the model: there is a certain reduction of angular motion and a dark fringe a bit more stable (fig 5, fig 6). We will continue to keep the new control under observation during the next data taking.