Upon reaching the site I found the ITF relocking. After a brief alignment of the PR/SR Mirrors, LOW_NOISE_3_SQZ was reached at the first attempt at 13:50 UTC, ITF back in Science Mode from 15:32 UTC.
At 15:07 UTC I set the ITF in Commissioning Mode to allow the planned ISC activity, DIFFp set point error signal test, carried out by Mantovani. The work could not be concluded due to an unlock at 15:22 UTC due to an earthquake (M 5.6 - 9 km W of Sulusaray, Turkey, 15:11 UTC). After various attempts we were able to reach LOW_NOISE_3_SQZ at 16:53 UTC, and the second planned activity, ENV Acustic Injection, started (see report #64024).
ENV Activity concluded at 18:11 UTC and ITF back in Science Mode. A quick adjusting period, from 18:16 UTC to 18:17 UTC, was done to allow the switch off the Speaker PDU.
The ITF unlocked at 19:34 UTC, relock in progress.

Guard tour (UTC):
17:06 - 17:44
19:03 - 19:30

Prompted by Michal, I verified if the occurrence of Scattered Light glitches from External Injection Bench motion associated with bad weather conditions has changed after the recent intervention and improvement of the EIB ground noise correction: #63763.

I examined the conditions during the last three main events of bad weather and Scattered Light glitches: February 25-26, March 26-27, and April 1-2.

Figure 1: time series of the V1:ENV_CEB_SEIS_W_50Hz_rms_0.1_1Hz during the three examined periods. In all three cases, the level of ground motion in the CEB is comparable.

Figures 2, 3 and 4: glitchgrams for Hrec during the three examined periods. The bottom time series represent the glitch rates for those triggers with frequency at peak between 20 and 2048 Hz, and SNR > 6.5. One interesting thing to notice is the presence of additional, very loud glitches with frequency at peak of about 90-100 Hz, associated with the last period of bad weather. Investigations are ongoing to understand their origin and if they are related or not to the ground motion.

Figures 5, 6 and 7: same as above but for the V1:LSC_SRCL channel, which in previous entries (#63761 and , #63420) was observed with Scattered Light glitches most correlated with the motion of the EIB, represented by the channel V1:SBE_EIB_GEO_H2_200Hz. In the period April 1-2, almost no glitches are present above 20 Hz, while in the other two periods, the rate is very high and correlated with the ground motion. The fact that a similar reduction in the rate of glitches is not visible in Hrec suggests a different origin for them.

Figure 8: a few Scattered Light glitches present in LSC_SRCL and their prediction with the SBE_EIB_GEO_H2_200Hz channel, as described in #63761.

Figure 9: the Scattered Light visible in Hrec is mostly explainable by the motion of the West End mirror. Using the technique described in #63761, the most correlated channel found is Sa_WE_F0_X_500Hz. In the attached text file, the list of all channels with correlation larger than 20%. The reconstruction is not perfect due to the sparseness and irregular shape of the glitches.

We performed a set of four acoustic injections in CEB with the AcousticInjectionCEB VPM process:

1) 17:07:55 UTC - (8 - 2000) Hz, 600 sec, 0.03 V

2) 17:21:05 UTC - (8 - 2000) Hz, 600 sec, 0.04 V  (25 min glitch at ~17:29:24 UTC)

3) 17:33:48 UTC - (8 - 2000 )Hz, 600 sec, 0.05 V

4) 18:04:40 UTC - (80 -100) Hz, 300 sec, 0.04 V

The directory path of output files is /virgoData/NoiseInjections/AcousticInjectionsO4/output. The list of file injections is reported below:

1) AcousticColored_NOISE_CEB_DetTerrace-1397495261.txt

2) AcousticColored_NOISE_CEB_DetTerrace-1397496052.txt

3) AcousticColored_NOISE_CEB_DetTerrace-1397496814.txt

4) AcousticColored_NOISE_CEB_DetTerrace-1397498666.txt

As Piernicola confirmed, the sidebands at 46.7 Hz and 53.3 Hz are the beating between 50 Hz and DIFFp_TX line at 3.3 Hz (purple curve).
Indeed, when the DIFFp_TX line was set to 4.8 Hz, there were two sidebands at 45.2 Hz and 54.8 Hz, respectively (blue curve).

This afternoon we performed a few tests on the OMC shutter with the ITF in NI single bounce in order to find a solution to the problem of OMC shutter getting stuck at the opening.

Fig.1 shows a test when sent a MOVETOLIMIT command via VPM to open the shutter. In this case the shutter motor get stuck, and we stopped it by sending the STOP MOTION command.

We have tested the opening of the shutter by sending some MOVEREL commands with a defined number of steps. We have found that with about 36000 steps the power on B1s reaches a plateau. Adding a bit of margin, we decided to perform 42000 steps to open the shutter. This is tested on Fig.2. We can see that it takes about 65 seconds to accomplish this motion.

We have then replaced the MOVETOLIMIT command by the MOVEREL command for the opening of the shutter in the class SHUTTER_OPENING of DET_MAIN. In the DET_MAIN.ini file, we have added the parameter nominal_open_steps = 42000, and we have updated the opening_duration to 80 seconds.

The opening and then closing of the shutter are tested with the automation on Fig.3. The closing is still performed with the MOVETOLIMIT command in order to be sure that we always start opening the shutter from the same position.

It was then possible to relock the ITF in Low Noise 3 at first trial.

The unlock due to bad weather was not associated to the limited range of the actuators, as happened other times. When the wind increased and reached 50 km/h in a few minutes, WE_MIR_CORR increased, but not enough to stay so close to the saturation (fig 1). The unlock happened during a clear drop of the sidebands, at the end of which some longitudinal loop was not very stable anymore (fig 2). The drop was associated to a misalignment of BS TY (fig 3) and PR_TY (fig 4). It is not clear to me which was the first which began to move, and what was the cause of this motion.

Upon arrival I found ITF in Science Mode.
ITF stable locked at LN3_SQZ (Science Mode) till 11:31 UTC when it unlocked due to the bad weather conditions. High wind activity.
According with Gouaty I set ITF in SINGLE_BOUNCE_NI in order to allow Romain on his investigation on OMC shutter and B1_PD3 noise.
TROUBLESHOOTING MODE set from 11:33 to 12:56 UTC. Activity concluded. Relock in progress...

NO DMS events Monitor to report
 

Looking at the spectrogram of Hrec, we observed a peak at 47.5 Hz which arose on April 15th, ~01:00 UTC (between two locks), see Figure 1 and 2.
It could be related to the F7 crossbar mode of the NI tower that we measured by magnetic injections to be at 47.65 Hz (VIR-0300A-24).
In this case, it seems that the line has shifted slightly in frequency.

Yesterday at 19:24 UTC the ITF unlocked because of a saturation of WE MIR actuation (fig 1). The correction was progressively increasing in the latest half an hour and it was often very close to 10 V in the latest minutes (fig 2). It was due to an increasing of ground motion between 0.5 and 1 Hz (fig 3), typically due to the weather close to the site (for instance, strong rain or particularly stormy sea). That frequency component was the main responsible of the large correction (fig 4); improving the filters at that frequency has for sure an impact at 1.8 Hz, so any change needs to be tested in terms of stability. There is also the chance that something can be improved at the level of the top stage control, but also in that case, the margin of improvement without worsening other frequency regions is small.

Transients are see also in DET B8 DC (Figure 1). Signal SBE_SWEB_F0_z_LVDT_500Hz converted into velocity and low pass filtered (fcut 0.5 Hz, in order to remove large high frequency noise in this signal) seems to reproduce sufficiently well the shape: Figure 2. Also the second order of arches (weakly seen in the photodiode)  is similarly reproduced doubling the velocity, although not plotted.  Maybe there is a better signal to use that measure the bench motion?

The attached Figure 1 shows the time when the OMC shutter was opened for the OMC lock and scan during the last maintenance. One can see a glitch on SDB1_OMC1_T1 indicating when the shutter starts to move. The power reaches a plateau on B1s about 16 seconds later. But, as already mentioned by Michal in https://logbook.virgo-gw.eu/virgo/?r=64012 , the line at 1721 Hz on the spectrum of SDB1_OMC1_T1 is present for about 20 minutes indicating that the shutter motor was still working during all this time. The line disappear between 06h39 and 06h40 utc (see Figure 2).

Figure 3 shows the closing of the OMC shutter. In this case we can see a first glitch on SDB1_OMC1_T1 when the shutter starts to move, and a second glitch about 35 seconds later when the shutter stops. The power on B1s reaches zero about 22 seconds after the first glitch. In this case the line at 1721 Hz on SDB1_OMC1_T1 disappear right after the second glitch, meaning that the shutter motor is properly stopped (Fig.4). Thus, it looks like the end limit sensor is still working for the shutter closing. The issue seems to be only for the opening.

Figure 1. Another channel that sees the noise is SDB1_B5_QD1_H (and all the other channels of the quadrants on SDB1). It sees the noise start when the slow shutter on SDB1 starts opening 15:59:16, then the noise level is constant, and the it stops. On B1 PD3 the noise level stops when it stops on B5 QD1, and it is present only when there is light on B1 PD3, so that is the reason why the level of noise on B1 PD3 varies in time during these 5 minutes.

This is most likely due to the slow shutter of the OMC which is mounted on an Agilis translation stage with electronic end of range stops. The stops are failing and the piezo-motor of the shutter keeps trying to push it further even though the translation stage arrived at the end of its range.

Figure 2. Shows that the OMC thermistor sees a line at ~1721 Hz when the slow shutter is in motion. This is likely the electrical cross-coupling of the voltage sent to the slow shutter showing up on the voltage readout used for the temperature measurement.  In blue is when the shutter moves and in purple when the shutter doesn't move. Unfortunately dataDisplay can't seem to handle the dynamic, and making an FFTtime plot of this channel doesn't seem to work.

FIgure 3. When the shutter is closing after an unlock there is also a 1721Hz line preens in OMC1 T1, but it is 10 times smaller.

Figure 4. OMC T1 also sees the 1721Hz when opening the OMC shutter in single bounce. The line stays on for about 30 minutes, before stopping by itself. The shutter is closed 20 minutes after the line stops by itself according to the shutter log file.

Action items to debug this further and mitigate it:

• To open the OMC slow shutter use a given number of step instead of moving the shutter to the limit. The needed number of steps to open the shutter need to be tested, for example in single bounce after the next unlock.
• Keep using the move to limit command for closing the shutter, so the shutter always goes back to the same position after a round trip.
• Test the operation of the OMC slow shutter in single bounce. The problem can be seen on OMC1 T1 when opening the shuter. Turn on/off the driver of the slow shutter and see if that restores the functioning of the translation stage stops.
• Investigate in past data if the shutter has started to slow down, if it takes more seconds between when the shutter command is sent and when the light appears on B1s.
• Check if the height of 1721Hz line is constant on OMC1 T1 when the shutter is opened, or if it changes in height after a fews tens of seconds when the translation stage arrives at the stop.
• Investigate in past data if the height of the 1721Hz line when the shutter is being opened has increased over the past months.

The first lock at 16h00-UTC was luky as  the OMC lock noise disappeared just after the OMC lock with B1_PD3_Audio (plot) but not the others ones  (ie 21h50m-UTC one)

Here we report a few more observations concerning the excess noise causing problems to reach DC readout.

The excess noise is visible on the audio channels of several photodiodes (Fig.1 and Fig.2): B1s, B1sP and B1_PD3 (which are on the same service mezzanine) as well as on the B1p_PD1 audio channel (however in this later case, only lines are present, while for B1s and B1_PD3 there is a clear impact on the noise floor).

As reported already by Diego, an extra noise is also present on the SDB1_LC_TX_err and SDB1_LC_TY_err signals at the same time as the noise is present on the photodiodes. This SDB1_LC_TX/TY_err signals are constructed from the SDB1 B5 QD2 H and V signals, which also see the extra noise. This observation concerning the SDB1 local control was already true yesterday (Fig.3)

Both the spectrum of the B1_PD1 audio channel, and those of SDB1_B5_QD2_H/V signals are clean when we are in CARM_NULL_1F (Fig.4).

The extra lines on B1p_PD1 and the extra noise on the B5 QD2 signals show up as soon as the OMC shutter is open (Fig.5). The lines showing up on B1_PD1 are at the same frequency as the lines seen on B1s (Fig.6), although the B1s noise floor changes when the lock of the OMC is acquired.

The following plots shows a 300s FFTTime of the SDB2_B1p_PD1_Audio, SDB2_B1_PD3_Audio and the  SDB2_B1s_PD1_Audio channels during the OMC lock acquisition where the purple rectangle highlights  the periods with the presence of unexpected lines

• successfull OMC lock
  • 2024-04-01-00h45m00-UTC : FFTTime plot
  • 2024-04-04-13h34m00-UTC:  FFTTime plot,
    • this FFT plot compare the ASD  of the related channels (Audio and sample) for 2 periods: when some lines are present(purple)  and a quiet period(blue}
  • 2024-04-07-19h41m00-UTC:  FFTTime plot
  • 2024-04-09-20h43m00-UTC:  FFTTime plot
    • this FFT plot compare the ASD  of the related channels (Audio and sample) for 2 periods: when some lines are present(purple)  and a quiet period(blue}
  • 2024-04-14-10h52m00-UTC:  FFTTime plot :
    • During the OMC temperature scan, there is a period (blue rectangle) where the lines amplitude increases in the SDB2_B1s_PD1_Audio channels and appears in the  SDB2_B1_PD3_Audio channel
    • this FFT plot compare the ASD  of the related channels (Audio and sample) for 2 periods: when some lines are present(purple) and a quiet period(blue}
  • 2024-04-16-00h09m00-UTC: FFTTime plot
    • same statement for the blue rectangle as the previous event
  • 2024-04-17-03h40m40-UTC: FFTTime  plot
    • same statement for the blue rectangle as the previous event
  • From all these plots it seems that
    • there is a noisy time period (purple rectangle) when the OMC shutter is opened . This time period has increased with the time  and seems to start to reduce (see last event)
    • when the noisy time period is long, during the OMC temperature scan  a more noisy period occurs (blue rectangle

• unsuccessfull OMC lock
  • 2024-04-16-21h24m20-UTC: FFTTime plot
    • the noise does not disappear but seems to increase according to the OMC temperature analysis (blue rectangle) to remain when the OMC is locked
    • this FFT plot and its zoom compare the ASD  of the related channels (Audio and sample) for 2 periods: at the beginning when the OMC shutter is opened (purple) and when the OMC is locked with B1_PD3 (blue) : the noise has increased when the OMC is locked with B1_PD3
  • 2024-04-17-00h14m00-UTC: FFTTime plot
  • For reasons unknown today, the noise present when opening the OMC shutter does not disappear and moreover increases triggering the unlocking of the OMC

Upon reaching the site, I found the ITF in LOW_NOISE_3_SQZ and in Science Mode. The ITF unlocked at 15:03 UTC due to an earthquake (M 6.3 - 17 km WSW of Uwajima, Japan, 14:14 UTC), preventing the lock of the infrared cavities.
From 15:40 UTC the influence of the shaking decreased enough and it was possible to relock.
ITF back in LOW_NOISE_3_SQZ at 16:16 UTC and the planned ISC/INJ activity (see report #64007) started; work concluded at 19:13 UTC.
The ITF unlocked at 19:24 UTC, followed by various unlocks at different steps of the locking acquisition (ACQUIRE_LOW_NOISE_2, LOCKING_CARM_NULL_1F).
Relock in progress.

No DMS alerts received were received during the shift.

Guard tour (UTC)
17:01 - 17:42
19:01 - 19:28

After the earthquake in Japan, we relocked at the first trial and after we arrived LN3_SQZ we started to tune the noise injections for the PSTAB loop (both at LF and HF). After the amplitude of the noise was tuned, we added the corresponding functions into the "injection_SSFS.py" script.

We then proceeded to repeat by hand the two last injections of the SSFS loop, that last shift made the ITF unlock. We decreased slightly their amplitude, while checking that the coherence with Hrec was still high. Now the script launches: CLEAN SSFS_UGF PSTAB_LF PSTAB_HF and then the 4 other SSFS injections at lower frequencies (down to 100 Hz). The operator launched the script and it worked without any problem. Figure 1 shows the injection corresponding to SSFS_UGF, and I think the current UGF is safe enough (I wouldn't push much higher, to ensure robustness, unless the noise injections suggest that it is limiting).

At this point we passed to the DIFFp topic. We started by opening the DIFFp_TX servo loop (which zeros the offset!), and then we started to put back the offset. However this didn't zeroed the new signals and so we continued to increase the offset. During this scan we tried to tune the phase, slightly, even if it was already quite good. Then we repeated the same operation with the TY loop. Figure 2 shows the two scans that we did. For the TY the phases were not clear so we left the deafult ones.

The phases we put are:DIFFp_TX_DCP_HF_mag_B1_phi0 = 0.5;       DIFFp_TX_DCP_mag_B1_phi0 =0.5

Today at about 15:30UTC, during the recovery operations of the interferometer, we reactivated the pumping station '200m' on the 'squeezing' vacuum line 

Today we had another event of the noise suddenly disappearing, and this happened before the lock of the OMC, at 16:02:59 UTC (Figure 1); the lock of OMC and DC Readout went through fine (Figure 2).

The error signal SDB1_LC_TY_err also shows a similar behaviour to B1_PD3 regarding the 1722 Hz line, but a much more broadband effect as well (Figure 1 and spectra in Figures 3 and 4).

For debugging purposes, the channels Audio and DC related to the B1_PD3 and B1s_PD1 photodiodes acquired at 100KHz are now stored with the "_FS" suffix and available in the RAW_FULL stream .

• operations complete at 2024-04-17-15h25m26-UTC

Below the list of the channels, acquired by the SDB2_DBOX_RightDown - SDB2_MezzPD2  mezzanines at 100KHz and stored with the  "_FS" suffix:

 V1:SDB2_B1_PD3_Audio_raw_FS              nData= 100000 min= -6.11e+03  mean=-4.89e+03  max=-3.56e+03 rate= 100000Hz
 V1:SDB2_B1_PD3_DC_raw_FS                 nData= 100000 min= -3.55e+03  mean=-3.31e+03  max=-3.07e+03 rate= 100000Hz
 V1:SDB2_B1sP_PD1_Audio_raw_FS            nData= 100000 min= -2.69e+05  mean=-2.67e+05  max=-2.65e+05 rate= 100000Hz
 V1:SDB2_B1sP_PD1_DC_raw_FS               nData= 100000 min= -3.62e+03  mean=-3.38e+03  max=-3.14e+03 rate= 100000Hz
 V1:SDB2_B1s_PD1_Audio_raw_FS             nData= 100000 min=  6.12e+04  mean=  6.2e+04  max= 6.28e+04 rate= 100000Hz
 V1:SDB2_B1s_PD1_DC_raw_FS                nData= 100000 min=  1.54e+04  mean= 1.57e+04  max=  1.6e+04 rate= 100000Hz

Are the transients corresponding to time that where outside of science mode during the same hour due to adjustments?

Yes for the first two, not completely for the third transient -- see attached plot. Are there some guidances to assess when switching back to science after one such adjustment?

Are the transients corresponding to time that where outside of science mode during the same hour due to adjustments? See figure 1.

These should be nice scattered light injections from moving SWEB https://logbook.virgo-gw.eu/virgo/?r=63997. It would be nice if someone could confirm it by looking at the same spectrogram for DET_B8_DC, and also see if the frequency corresponds to the speed of the bench in the Z direction (along the beam).

Continuing the investigation of the spectral noise on the first week of O4b data, we found an interesting structure of "wandering sidebands" around the 50 Hz line. (S

Figure 1 shows the spectrogram in the region 42-58 Hz around the frequency of the electric mains. Two pairs of wandering sidebands are visible at 44 (Figure 2) and 56 Hz, and at 46.7 (Figure 3) and 53.3 Hz (Figure 5).

The oscillations in frequency, especially those of the 44 and 56 Hz lines, are the same as those of the 50 Hz line (Figure 4) but enlarged by many (~20) times: ~0.50 Hz vs ~0.025 Hz

Additionally, taking advantage of a couple of glitches present in the analyzed data period (the vertical lines), we notice that the 44 and 47.3 Hz lines are synchronous with the 50 Hz line. Also, the 53.3 Hz line seems to have the same phase, while the one at 56 Hz is flipped vertically: Figure 5.

The search for bilinear couplings with MONET highlighted that the 46.7 and 53.3 Hz sidebands are produced by the modulation of the mains line (ENV_CEB_UPS_VOLT_R_2000Hz) and the ASC_DIFFp_T{X,Y} angular control channels: Figure 6 and text file.

A barely negligible coherence has been found for the lines at 44 and 56 Hz; just a 0.1 with SDB1_LC_TX at 56 Hz: Figure 7. This could be a consequence of the large variations in the frequency of the line, which dilute the cohere over multiple frequency bins and, averaged gets small.

Of course, the same structure is present around the other harmonics of the 50 Hz line.

Upon arrival I found ITF in Science Mode (LN3_SQZ).
At ~ 08:50 UTC the vertical position loop of SWEB opened. I set Adjusting Mode and closed it but it opened again after few minutes. I tried to act on MotSwitch but it was not working. I contacted SBE OnCall but the issue was related to the old version of MotSwitch VPM config file. I restarted the MotSwitch process and act on MotSwitch Olwin tool to recover the vertical position of the bench. I made a check for all SBE MotSwitch processes are running with the v4r4 version.
Science Mode set again at 09:45 UTC.  

No DMS Events Monitor to report

DET
(16-04-2024 19:30 - 23:15) From remote
Status: On-going
Description: unlock at DC_READOUT step

SBE
(17-04-2024 10:00 - 17-04-2024 10:30) Operator on site with expert from remote
Status: Ended
Description: SWEB vertical loop open.
Problem fixed thanks to Carbognani who investigated on MotSwitch VPM config file process.