I have seen the interferometer and injection unlocked. The INJ MAIN automation was going through an endless loop passing through the DOWN state. I have put both ITF LOCK and INJ MAIN into DOWN.
Figure 1. There seems to be a major issue with the laser amplifier cooling. The temperature railed at 45 degrees.
Figure 1. After the recovey of the amplifier water cooling, the power in transmission of the PMC is fluctuating much more and the power reflected by the PMC is a factor 2 larger. The beam alignment into the PMC may need to be adjusted. This is very likely the reason for the glitches in the sensitivity since last night.
UPV + VetoPerf list several INJ signals as the main witnesses for these loud glitches: https://scientists.virgo-gw.eu/DataAnalysis/DetCharDev/users/narnaud/UPV/20250102/V1:Hrec_hoft_16384Hz/perf/vp.html.
V1:SIB2_B2_DC, vetoed clusters: 157 (66.245 %)
V1:SIB2_RFC_PD1_DC, vetoed clusters: 151 (63.713 %)
V1:SIB2_RFC_QD2_H, vetoed clusters: 151 (63.713 %)
V1:ASC_B2_QD2_Sum, vetoed clusters: 151 (63.713 %)
V1:SIB2_RFC_PD2_DC, vetoed clusters: 150 (63.291 %)
V1:INJ_IMC_TRA_DC, vetoed clusters: 149 (62.869 %)
V1:INJ_ITF_INPUT, vetoed clusters: 149 (62.869 %)
V1:INJ_IBJM_DC_H, vetoed clusters: 142 (59.916 %)
V1:BsX_QN_DC, vetoed clusters: 142 (59.916 %)
V1:INJ_IBJM_DC_H_e, vetoed clusters: 142 (59.916 %)
An example before / after, using the channel V1:SIB2_B2_DC as a veto for those glitches.
These loud glitches are likely to impact the quality of the Virgo Science data and may prevent using them in case of a low-latency trigger.
The change of temperature of the neovan head has usually a large impact on alignment of the beam sent to the PMC (and a bit less on mismatch). It is likely that the thermal drift will continue for order of few hours before reaching a steady state, I suggest to check the alignment of the beam sent to the PMC only after being sure that the thermal state is stable.
The bad weather did not help to accumulate data but the two plots in attachment show
1) the Omicron trigger distribution in the time-frequency plane;
2) the Omicron trigger rates for different SNR cuts.
Both plots start a day before the INJ problem to show more clearly the difference before/after,
Although bad weather glitches started to appear around 1500 UTC, it is not clear that today's work improved the situation glitch-wise. While the glitch rate seems to have decreased slowly in the morning, during the ~10 hours Virgo was locked.
## I re-post the entry of the afternoon which I found blocked into the draft ##
Yesterday afternoon the Neovan head Chiller suddenly stopped. As a result, the temperature of the head rapidely increased, the power amplified gradually decreased, the ITF unlocked and the INJ metatrone node entered in a bad loop (see fig.1). Even if the the neovan temperature shows a plateau, the real temperature went far higher ( most probably between 90⁰ and 100⁰C) without triggering any protection mode of the amplifier, as it happens for the neovan electronics failure.
From remote, I decreased the current of the pumping diodes and switched off the amplifier. The very high temperature reached by the head is confimed by the time it needed to recover the room temperature after the swich off (see fig. 2).
Once onsite, I tried to restart the faulty chiller without success. Thus I decided to replace it with one additional spare we had in Optics Lab (see fig. 3). Once the spare chiller tested and replaced, I could restart the amplifier. Beside, I tested the original chiller on a closed water circuit, and it appeared that there was some dirt/mug in the pump (fig. 4). I left it working for some time and it seems to work correctly now.
Once the chiller ON, I switched back on the amplifier with the usual current settings (diode 1-2 @4.7A, diode 3-4 @5.3A), however the output amp beam didn't recover the same, either in term of power and alignment. The reason is most probably that, due to the very high temperature reached by the crystal, something changed inside the head (crystal glue, something else?). The misalignment is confirmed also by the two quadrants (in fig. 5 the normalised values) and from the camera (fig. 6) at the output of the amplifier.
By operating on the current of the diodes and on the temperature of the neovan head (with the temperature of the chiller), I managed to increase the matching towards the PMC in order to reach the minimum power needed to get a stable output from the PMC and IMC @18W. After the intervention, the ITF could lock up to LN3 at first attempt.
However, changing the pump power does not affect (or, only marginally) the misalignment of the beam. In particular, the beam is misaligned also into the AOM, the actuator of the PSTAB loop, which most probably is tha reason of the saturation of the PSTAB correction at low frequency, which eventually causes the power glitches visibles on Hrec.
To solve, at least partially, the problem on the PSTAB, a realignment of the beam entering the Neovan.
On Friday morning, I continued the recovery of the INJ/PSL system.
In agreement with Walid, we decided to intervene directly the laser bench.
Entering the Laser Lab, I firstly inspected the state of the Neovan head after the chiller failure and I found a tiny water leakage from one of the chiller pipe (fig. 1-2), probably the plastic joint in contact with the box degraded due to the high temperature.
For the time being, I didn't try to tighten it in order to avoid any further problem with the alignment. However, even if the leakage doesn't seems to be big, we will try to solve it in the next days.
Then, I started to work on the alignment of the beam into the Neovan by moving the steering mirror right at the entrance, looking at the QDs position as reference. Unfortunately, while I could recover quite easily the horizontal position, I struggled to come back to the previous vertical position without completely loose the matching into the amplifier (the output AMP power drop by more the 50%, see fig. 3). Most probably, the failure of the Jan 1st caused some misalignment of the crystals into the head, that cannot be recover by a simple tilt of the beam.
Also, in this condition I tried anyway to enter the PMC with higher power by using the IPC0 in front of the AOM.
I could relock the IMC and the ITF, however without any significant improvement on the PSTAB loop.
Thus, we decided to operate differently. I first played with the pump currents and the input beam position in order to recover a decent matching into the amplifier as close as possible to the original position on the QDs (fig. 5).
Then, with only the mirror in front of the AOM, I improved the matching into both the PMC and AOM, mainly trying to minimize the HOM of the PMC (with it in scanning mode). In this way, I could minimize both the alignment and the matching (orders 1 and 2). I slightly increase the IPC0 (fig. 4, 80.5 vs 78 before) in order to keep more or less the same power and shape at the ouput of the PMC.
The effect on the PSTAB loop was positive too, with almost the same noise rms level as before the failure seen by the out of loop PSTAB photodiode (PSTAB_PDa_AC_monit).
Since last friday, the ITF could stay locked with good duty cicle and ~54Mpc mean sensitivity. Apparently, no further PSTAB glitches have been seen on the sensitivity.
Today, we went in Laser Lab to inspect again the Neovan head. As we can see in fig. 1, we found almost no water on the bench. Probably some particles in the water and the cooler temperature of the head helped in limiting the small leakage.
However, for matter of security, we left a small plastic container to limit any possible further leakage, at least untill we solve the problem (fig. 2).
