Yesterday at 19:24 UTC the ITF unlocked because of a saturation of WE MIR actuation (fig 1). The correction was progressively increasing in the latest half an hour and it was often very close to 10 V in the latest minutes (fig 2). It was due to an increasing of ground motion between 0.5 and 1 Hz (fig 3), typically due to the weather close to the site (for instance, strong rain or particularly stormy sea). That frequency component was the main responsible of the large correction (fig 4); improving the filters at that frequency has for sure an impact at 1.8 Hz, so any change needs to be tested in terms of stability. There is also the chance that something can be improved at the level of the top stage control, but also in that case, the margin of improvement without worsening other frequency regions is small.

Transients are see also in DET B8 DC (Figure 1). Signal SBE_SWEB_F0_z_LVDT_500Hz converted into velocity and low pass filtered (fcut 0.5 Hz, in order to remove large high frequency noise in this signal) seems to reproduce sufficiently well the shape: Figure 2. Also the second order of arches (weakly seen in the photodiode)  is similarly reproduced doubling the velocity, although not plotted.  Maybe there is a better signal to use that measure the bench motion?

The attached Figure 1 shows the time when the OMC shutter was opened for the OMC lock and scan during the last maintenance. One can see a glitch on SDB1_OMC1_T1 indicating when the shutter starts to move. The power reaches a plateau on B1s about 16 seconds later. But, as already mentioned by Michal in https://logbook.virgo-gw.eu/virgo/?r=64012 , the line at 1721 Hz on the spectrum of SDB1_OMC1_T1 is present for about 20 minutes indicating that the shutter motor was still working during all this time. The line disappear between 06h39 and 06h40 utc (see Figure 2).

Figure 3 shows the closing of the OMC shutter. In this case we can see a first glitch on SDB1_OMC1_T1 when the shutter starts to move, and a second glitch about 35 seconds later when the shutter stops. The power on B1s reaches zero about 22 seconds after the first glitch. In this case the line at 1721 Hz on SDB1_OMC1_T1 disappear right after the second glitch, meaning that the shutter motor is properly stopped (Fig.4). Thus, it looks like the end limit sensor is still working for the shutter closing. The issue seems to be only for the opening.

Figure 1. Another channel that sees the noise is SDB1_B5_QD1_H (and all the other channels of the quadrants on SDB1). It sees the noise start when the slow shutter on SDB1 starts opening 15:59:16, then the noise level is constant, and the it stops. On B1 PD3 the noise level stops when it stops on B5 QD1, and it is present only when there is light on B1 PD3, so that is the reason why the level of noise on B1 PD3 varies in time during these 5 minutes.

This is most likely due to the slow shutter of the OMC which is mounted on an Agilis translation stage with electronic end of range stops. The stops are failing and the piezo-motor of the shutter keeps trying to push it further even though the translation stage arrived at the end of its range.

Figure 2. Shows that the OMC thermistor sees a line at ~1721 Hz when the slow shutter is in motion. This is likely the electrical cross-coupling of the voltage sent to the slow shutter showing up on the voltage readout used for the temperature measurement.  In blue is when the shutter moves and in purple when the shutter doesn't move. Unfortunately dataDisplay can't seem to handle the dynamic, and making an FFTtime plot of this channel doesn't seem to work.

FIgure 3. When the shutter is closing after an unlock there is also a 1721Hz line preens in OMC1 T1, but it is 10 times smaller.

Figure 4. OMC T1 also sees the 1721Hz when opening the OMC shutter in single bounce. The line stays on for about 30 minutes, before stopping by itself. The shutter is closed 20 minutes after the line stops by itself according to the shutter log file.

Action items to debug this further and mitigate it:

• To open the OMC slow shutter use a given number of step instead of moving the shutter to the limit. The needed number of steps to open the shutter need to be tested, for example in single bounce after the next unlock.
• Keep using the move to limit command for closing the shutter, so the shutter always goes back to the same position after a round trip.
• Test the operation of the OMC slow shutter in single bounce. The problem can be seen on OMC1 T1 when opening the shuter. Turn on/off the driver of the slow shutter and see if that restores the functioning of the translation stage stops.
• Investigate in past data if the shutter has started to slow down, if it takes more seconds between when the shutter command is sent and when the light appears on B1s.
• Check if the height of 1721Hz line is constant on OMC1 T1 when the shutter is opened, or if it changes in height after a fews tens of seconds when the translation stage arrives at the stop.
• Investigate in past data if the height of the 1721Hz line when the shutter is being opened has increased over the past months.

The first lock at 16h00-UTC was luky as  the OMC lock noise disappeared just after the OMC lock with B1_PD3_Audio (plot) but not the others ones  (ie 21h50m-UTC one)

Here we report a few more observations concerning the excess noise causing problems to reach DC readout.

The excess noise is visible on the audio channels of several photodiodes (Fig.1 and Fig.2): B1s, B1sP and B1_PD3 (which are on the same service mezzanine) as well as on the B1p_PD1 audio channel (however in this later case, only lines are present, while for B1s and B1_PD3 there is a clear impact on the noise floor).

As reported already by Diego, an extra noise is also present on the SDB1_LC_TX_err and SDB1_LC_TY_err signals at the same time as the noise is present on the photodiodes. This SDB1_LC_TX/TY_err signals are constructed from the SDB1 B5 QD2 H and V signals, which also see the extra noise. This observation concerning the SDB1 local control was already true yesterday (Fig.3)

Both the spectrum of the B1_PD1 audio channel, and those of SDB1_B5_QD2_H/V signals are clean when we are in CARM_NULL_1F (Fig.4).

The extra lines on B1p_PD1 and the extra noise on the B5 QD2 signals show up as soon as the OMC shutter is open (Fig.5). The lines showing up on B1_PD1 are at the same frequency as the lines seen on B1s (Fig.6), although the B1s noise floor changes when the lock of the OMC is acquired.

The following plots shows a 300s FFTTime of the SDB2_B1p_PD1_Audio, SDB2_B1_PD3_Audio and the  SDB2_B1s_PD1_Audio channels during the OMC lock acquisition where the purple rectangle highlights  the periods with the presence of unexpected lines

• successfull OMC lock
  • 2024-04-01-00h45m00-UTC : FFTTime plot
  • 2024-04-04-13h34m00-UTC:  FFTTime plot,
    • this FFT plot compare the ASD  of the related channels (Audio and sample) for 2 periods: when some lines are present(purple)  and a quiet period(blue}
  • 2024-04-07-19h41m00-UTC:  FFTTime plot
  • 2024-04-09-20h43m00-UTC:  FFTTime plot
    • this FFT plot compare the ASD  of the related channels (Audio and sample) for 2 periods: when some lines are present(purple)  and a quiet period(blue}
  • 2024-04-14-10h52m00-UTC:  FFTTime plot :
    • During the OMC temperature scan, there is a period (blue rectangle) where the lines amplitude increases in the SDB2_B1s_PD1_Audio channels and appears in the  SDB2_B1_PD3_Audio channel
    • this FFT plot compare the ASD  of the related channels (Audio and sample) for 2 periods: when some lines are present(purple) and a quiet period(blue}
  • 2024-04-16-00h09m00-UTC: FFTTime plot
    • same statement for the blue rectangle as the previous event
  • 2024-04-17-03h40m40-UTC: FFTTime  plot
    • same statement for the blue rectangle as the previous event
  • From all these plots it seems that
    • there is a noisy time period (purple rectangle) when the OMC shutter is opened . This time period has increased with the time  and seems to start to reduce (see last event)
    • when the noisy time period is long, during the OMC temperature scan  a more noisy period occurs (blue rectangle

• unsuccessfull OMC lock
  • 2024-04-16-21h24m20-UTC: FFTTime plot
    • the noise does not disappear but seems to increase according to the OMC temperature analysis (blue rectangle) to remain when the OMC is locked
    • this FFT plot and its zoom compare the ASD  of the related channels (Audio and sample) for 2 periods: at the beginning when the OMC shutter is opened (purple) and when the OMC is locked with B1_PD3 (blue) : the noise has increased when the OMC is locked with B1_PD3
  • 2024-04-17-00h14m00-UTC: FFTTime plot
  • For reasons unknown today, the noise present when opening the OMC shutter does not disappear and moreover increases triggering the unlocking of the OMC

Upon reaching the site, I found the ITF in LOW_NOISE_3_SQZ and in Science Mode. The ITF unlocked at 15:03 UTC due to an earthquake (M 6.3 - 17 km WSW of Uwajima, Japan, 14:14 UTC), preventing the lock of the infrared cavities.
From 15:40 UTC the influence of the shaking decreased enough and it was possible to relock.
ITF back in LOW_NOISE_3_SQZ at 16:16 UTC and the planned ISC/INJ activity (see report #64007) started; work concluded at 19:13 UTC.
The ITF unlocked at 19:24 UTC, followed by various unlocks at different steps of the locking acquisition (ACQUIRE_LOW_NOISE_2, LOCKING_CARM_NULL_1F).
Relock in progress.

No DMS alerts received were received during the shift.

Guard tour (UTC)
17:01 - 17:42
19:01 - 19:28

After the earthquake in Japan, we relocked at the first trial and after we arrived LN3_SQZ we started to tune the noise injections for the PSTAB loop (both at LF and HF). After the amplitude of the noise was tuned, we added the corresponding functions into the "injection_SSFS.py" script.

We then proceeded to repeat by hand the two last injections of the SSFS loop, that last shift made the ITF unlock. We decreased slightly their amplitude, while checking that the coherence with Hrec was still high. Now the script launches: CLEAN SSFS_UGF PSTAB_LF PSTAB_HF and then the 4 other SSFS injections at lower frequencies (down to 100 Hz). The operator launched the script and it worked without any problem. Figure 1 shows the injection corresponding to SSFS_UGF, and I think the current UGF is safe enough (I wouldn't push much higher, to ensure robustness, unless the noise injections suggest that it is limiting).

At this point we passed to the DIFFp topic. We started by opening the DIFFp_TX servo loop (which zeros the offset!), and then we started to put back the offset. However this didn't zeroed the new signals and so we continued to increase the offset. During this scan we tried to tune the phase, slightly, even if it was already quite good. Then we repeated the same operation with the TY loop. Figure 2 shows the two scans that we did. For the TY the phases were not clear so we left the deafult ones.

The phases we put are:DIFFp_TX_DCP_HF_mag_B1_phi0 = 0.5;       DIFFp_TX_DCP_mag_B1_phi0 =0.5

Today at about 15:30UTC, during the recovery operations of the interferometer, we reactivated the pumping station '200m' on the 'squeezing' vacuum line 

Today we had another event of the noise suddenly disappearing, and this happened before the lock of the OMC, at 16:02:59 UTC (Figure 1); the lock of OMC and DC Readout went through fine (Figure 2).

The error signal SDB1_LC_TY_err also shows a similar behaviour to B1_PD3 regarding the 1722 Hz line, but a much more broadband effect as well (Figure 1 and spectra in Figures 3 and 4).

For debugging purposes, the channels Audio and DC related to the B1_PD3 and B1s_PD1 photodiodes acquired at 100KHz are now stored with the "_FS" suffix and available in the RAW_FULL stream .

• operations complete at 2024-04-17-15h25m26-UTC

Below the list of the channels, acquired by the SDB2_DBOX_RightDown - SDB2_MezzPD2  mezzanines at 100KHz and stored with the  "_FS" suffix:

 V1:SDB2_B1_PD3_Audio_raw_FS              nData= 100000 min= -6.11e+03  mean=-4.89e+03  max=-3.56e+03 rate= 100000Hz
 V1:SDB2_B1_PD3_DC_raw_FS                 nData= 100000 min= -3.55e+03  mean=-3.31e+03  max=-3.07e+03 rate= 100000Hz
 V1:SDB2_B1sP_PD1_Audio_raw_FS            nData= 100000 min= -2.69e+05  mean=-2.67e+05  max=-2.65e+05 rate= 100000Hz
 V1:SDB2_B1sP_PD1_DC_raw_FS               nData= 100000 min= -3.62e+03  mean=-3.38e+03  max=-3.14e+03 rate= 100000Hz
 V1:SDB2_B1s_PD1_Audio_raw_FS             nData= 100000 min=  6.12e+04  mean=  6.2e+04  max= 6.28e+04 rate= 100000Hz
 V1:SDB2_B1s_PD1_DC_raw_FS                nData= 100000 min=  1.54e+04  mean= 1.57e+04  max=  1.6e+04 rate= 100000Hz

Are the transients corresponding to time that where outside of science mode during the same hour due to adjustments?

Yes for the first two, not completely for the third transient -- see attached plot. Are there some guidances to assess when switching back to science after one such adjustment?

Are the transients corresponding to time that where outside of science mode during the same hour due to adjustments? See figure 1.

These should be nice scattered light injections from moving SWEB https://logbook.virgo-gw.eu/virgo/?r=63997. It would be nice if someone could confirm it by looking at the same spectrogram for DET_B8_DC, and also see if the frequency corresponds to the speed of the bench in the Z direction (along the beam).

Continuing the investigation of the spectral noise on the first week of O4b data, we found an interesting structure of "wandering sidebands" around the 50 Hz line. (S

Figure 1 shows the spectrogram in the region 42-58 Hz around the frequency of the electric mains. Two pairs of wandering sidebands are visible at 44 (Figure 2) and 56 Hz, and at 46.7 (Figure 3) and 53.3 Hz (Figure 5).

The oscillations in frequency, especially those of the 44 and 56 Hz lines, are the same as those of the 50 Hz line (Figure 4) but enlarged by many (~20) times: ~0.50 Hz vs ~0.025 Hz

Additionally, taking advantage of a couple of glitches present in the analyzed data period (the vertical lines), we notice that the 44 and 47.3 Hz lines are synchronous with the 50 Hz line. Also, the 53.3 Hz line seems to have the same phase, while the one at 56 Hz is flipped vertically: Figure 5.

The search for bilinear couplings with MONET highlighted that the 46.7 and 53.3 Hz sidebands are produced by the modulation of the mains line (ENV_CEB_UPS_VOLT_R_2000Hz) and the ASC_DIFFp_T{X,Y} angular control channels: Figure 6 and text file.

A barely negligible coherence has been found for the lines at 44 and 56 Hz; just a 0.1 with SDB1_LC_TX at 56 Hz: Figure 7. This could be a consequence of the large variations in the frequency of the line, which dilute the cohere over multiple frequency bins and, averaged gets small.

Of course, the same structure is present around the other harmonics of the 50 Hz line.

Upon arrival I found ITF in Science Mode (LN3_SQZ).
At ~ 08:50 UTC the vertical position loop of SWEB opened. I set Adjusting Mode and closed it but it opened again after few minutes. I tried to act on MotSwitch but it was not working. I contacted SBE OnCall but the issue was related to the old version of MotSwitch VPM config file. I restarted the MotSwitch process and act on MotSwitch Olwin tool to recover the vertical position of the bench. I made a check for all SBE MotSwitch processes are running with the v4r4 version.
Science Mode set again at 09:45 UTC.  

No DMS Events Monitor to report

DET
(16-04-2024 19:30 - 23:15) From remote
Status: On-going
Description: unlock at DC_READOUT step

SBE
(17-04-2024 10:00 - 17-04-2024 10:30) Operator on site with expert from remote
Status: Ended
Description: SWEB vertical loop open.
Problem fixed thanks to Carbognani who investigated on MotSwitch VPM config file process.

This morning, three transient noises, lasting several tens of seconds and limited to the 10-100 Hz frequency band, occured around 9h08 UTC, 9h38 UTC, 9h45 UTC.
Plot1 shows them in the spectrograms of LSC_DARM and Hrec_hoft_raw (on the same plot we can see also two 25' glitches).
Plot2 is a zoom on the third of those transient noises.

In the framework of the ongoing investigations for the study of spectral noise, I repeated the analysis done for the mini-Engineering Run, with the estimate of a high-resolution sensitivity curve and the rough identification of the main spectral lines, as described in this logbook entry: #63221. The result can be used as a reference and for comparison with the output of more refined tools, such as NoEMi.

As a remark, the frequency resolution of 1mHz is obtained with 1000 second-long FFTs. This duration is constrained by the 25-minute glitches; estimating the ASD with median averages is robust to these glitches if most of the FFTs are not affected by their presence. This requires using FFTs shorter than 25 minutes (1500 sec).

The data segment to be analyzed was the one with the longest duration in Science mode, weighted by the lowest coefficient of variation. The choice fell on the segment: 2024-04-13 01:53:18 to 2024-04-13 18:30:04 (59806.0 sec).

I attach to this entry the pdf with the details of the various frequency regions of the spectrum and the CSV with the identified lines. I plan to repeat this analysis for the next weeks, and attach the results in this git issue and its sub-tasks, for reference.

I'm adding also the following images:

Figure 1: high-resolution sensitivity curve with red lines corresponding to the identified spectral lines. Refer to the PDF file for the details.

Figure 2: detail of the 970-995 Hz region, showing a comb of lines spaced about ~2.83 Hz (0.35 sec).

Figure 3: spectrogram of the 955-1005 Hz region, showing the wandering line already documented in these logbook entries: #60093, #61873.

As of today online BruCo should run daily on Science segments -- until now it was running on locked segments. The segments processed by online BruCo should thus be more stable, while they may also be more numerous -- as one can leave Science for a short while w/o loosing the lock. I'll monitor the potential increase of running time in the coming days. Another improvement is that loud glitches (defined as having an Omicron SNR greater than 50) should now be excluded from the segments processed by online Bruco. Online BruCo is configured to run of segments lasting at least 15 minutes: thus, the 25-minute glitches should be safely removed w/o rejecting Science segments. I'll keep an eye on that as well to make sure the loud glitch removal doesn't kill all the online BruCo input segments.

After the modification of PR ACC H modulation frequency, we had 3 fast glitches - the usual rate observed recently - but no one triggered a trip. This is a good news: it seems that the problem has been fixed. To be noticed that the trips disappeared also from the vertical loop, but PR ACC V modulation frequency was not touched.

ITF found in Commissioning mode for: Change of logic for the LOW_NOISE_1 and LOW_NOISE_2 to bypass LowNoise1 actuation stage; activity concluded at 21:00 UTC

In parallel, Romain and Victor were investigating the excess noise on B1_PD3 that was preventing to lock the OMC thus at 21:15 I set Troubleshooting.

At 23:14 UTC we could lock the OMC but we unlocked during the transition to LN2; unfortunately the excess noise came back after the previous unlock.

Diego worked up to 1:00 UTC trying to understand the problem but we kept unlocking during the lock of the OMC; for the next lock acquisition we decided to wait more time in CARM_NULL_1F and try to perform the trick of the SR but unfortunately the ITF unlocked again.

The lock of the OMC kept failing for the next two hours; then at, around 3:00 UTC, the OMC locked again but the ITF unlocked LN2.

At the next attempt the ITF reachedn LN3_SQZ thus at 4:01 UTC I could set Science mode.

Guard tous (time in UTC)

21:00-21:30; 23:00 -23:30; 1:00-1:30; 3:!0-3:40

DET
(16-04-2024 19:30 - 23:15) From remote
Status: On-going
Description: unlock at DC_READOUT step

Unfortunately the excess noise came back already at the following lock acquisition, with the same features (Fig. 1, purple, red and blue are noise, no noise, and noise again respectively).

Looking at the B1_PD3_DC spectrum we can see many big lines (and sidebands) (Fig. 2); the most offending in terms of SNR is the one at ~6885 Hz (Fig. 3), but what is striking at a first glance is that all of them are separated by ~1722 Hz, although the lowest one is not the strongest one, so maybe also some aliasing is involved.

As part of the today's commissioning work on the MIR/MAR reallocation between IMs and EMs, the logic of the two LowNoise configurations has been changed: a new parameter, bypass_ln1, is defined as boolean in the [ITF_LOCK] section of the ITF_LOCK.ini file.

When this parameter is set to True, during the LOW_NOISE_1 transition we will keep using the Input Mirrors to actuate. And obviously nothing will be done on the relays. Also, we will prepare the End Mirrors to use directly their LowNoise configuration, i.e. LowNoise2.

During the LOW_NOISE_2 transition, we will jump directly from HighPower to LowNoise2. The only caveat here is whether we can survive acting on the IM relays when already in LOW_NOISE_2 (we usually do that in LOW_NOISE_1). The other caveat is that with the very high useism we have now, the standard lock acquisition would probably not work anyway given our current problems.

More information on the changes at the level of the actuators are reported in the aforementioned entry.

After the (unfortunately temporary) fix of DC Readout we managed to test the new logic described here; unfortunately, as feared, acting on the IM relays while already in LowNoise2 made the ITF to unlock, at least in the single test that we could make (unlock at 23:19:51 UTC); this is apparently strange, as this kind of action was done in the past for the charge measurements, but it could be that the full switch on both IMs at the same time is too much. We could think of acting on them in series on one IM at a time, to be studied.

After this test, and seeing that the useism is going down (still far from being low, though), we reverted to the standard strategy, which needs a check anyway since it is embedded in the same new logic. Unfortunately we started to have issues again in the OMC/DC Readout phase, so this is still ongoing at the time of writing.