Yesterday, 2023-09-22 around 14h30 UTC, I have restarted Hrec with a config modified as the following:
- The thresholds on the SNR of calibration lines to be used for optical response adjustement has been moved from 3 to 0 for BS, NE and WE (it was already 0 for PR).
- The channel LSC_SRCL has been added in the noise subtraction in the frequency band 8-40 Hz.

Hrec uses the classical noise subtraction (that we call dTF). I did the same modifications on HrecMTX which uses the new noise subtraction (that we call Mtx).
Plot1 upper plot: compares the BNS range for hoft_raw (before noise subtraction) and hoft (after noise subtraction) for dTF (orange) and Mtx (green)
Plot1 lower plot: show the same but only for dTF and in a time period where LSC_SRCL was not yet included in the noise subtraction.

Plot2,3,4 show the coherence between LSC_MICH, LSC_SRCL and the hoft (of dTF or Mtx) in the time period where LSC_SRCL was included in the noise subtraction.
It shows frequency bands around 20 Hz, 160 Hz and 3000 Hz respectively.
We can see that the coherence between MICH and SRCL is mainly below 10 Hz. Thus the security provided by Mtx against problems coming from overlap of the frequency bands of noise subtraction for SRCL (8-40 Hz) and for MICH (8-200 Hz) is not really needed. Keeping Hrec and HrecMTX running for a few days may confirm this.

The situation seems improved up to CARM Null (but the wind also went down); instead, after DC Readout the B1 PDs keep saturating (Figures 1 and 2), also after the LOW_NOISE_1 transition (where they are forced to use DC), so during the LOW_NOISE_2 transition and in steady state (as in the 1 minute lock). As a test I completely disabled the saturation switch, so it remains forced to 1 and the B1 PDs will keep using the DC and not the Audio information. First test shortly in the current acquisition.

FOLLOW-UP: also this did not work, B1 consistently on DC but glitches on DARM and correction saturation. SR alignment gets spoiled (because of the DARM glitches?), and I don't know how the OMC reacts to glitches on B1_PD2 (Figure 3); DARM_UGF not informative as coherence too low, DARM_OG really low.

Following a phone call by the operator we checked the situation regarding the saturation of the audio channels of the B1 photodiodes during the transition from DC readount to Low Noise 2. An example of such a transition during this afternoon is shown on the attached figure.

First we can see that the saturation switch is kept to 1 all the way from DC readout to Low Noise 2, as expected since Diego implemented this in the automation. This means that the blended signals are built from only the DC signals during all this phase, as we would wish since the audio channels are saturating most of the time.

We were also wondering if the glitches seen in the DC channels around 16h26 (when we acquire Low Noise 2) could be induced by the saturation of the audio channel anyway (by some electronic cross-talk). One piece of answer comes from the observation of the period between 16h20 and 16h24 utc, when the audio channels are permanently saturating. At that time there are no glitches in the DC channels. We may therefore conclude that the fact that the audio channel saturates is not responsible for the glitches observed in the DC channels (whose cause remains therefore not understood, it can be some glitches in the interferometer).

Upon arrival I found ITF locked at LN3 since 4h. Unfortunately it unlocked and I spent all the shift to try to acheive a stable lock.
The microseism due to the sea activity (fig1) prevented the lock for the whole shift. 
ITF locked in LN2 at 13:57 UTC. For 1 minute.
Etalon tuning not performed.

SBE
After the unlock of 06:00 UTC, SIB2 vertical position drifted. I recover it via stepping motors.

ISC
(23-09-2023 10:00 - 23-09-2023 12:00) From remote
Status: Ended
Description: ITF lock instability

I gave a look at the unlocks from this morning, both during lock acquisition and in the "steady" states; besides a couple of them, they fall in three categories:

• unlocks during the CARM handoff to the IMC: long standing story, they have been quite reduced in frequency after the last change in CARM offset at the time of the handoff, but it looks like that bad weather (as in wind and sea activity, which today are both annoying) has an impact on those;
• unlocks at apparently steady state in CARM_NULL_1F: "apparently" as they happen just a few seconds after getting there; I could find one definite instance, and two other suspect ones, of the fact that they are induced by the engagement of the CARM2RFC loop, which either saturates (Figure 1) and/or pushes a lot with "fast" transients +-5/6 V (Figures 2 and 3); for the time being I tried using a different CARM filter (1 success out of 2 at the time of writing), I will observe during the day and maybe change strategy (lower gain to act more as a drift control to avoid fast/big corrections);
• unlocks in LOW_NOISE_2: a couple of them happened (Figures 4 to 7, two unlocks) with a quickly diverging 55 Hz oscillation, most probably induced by the diagonalization process which abruptly changes sign to a couple of contributions; I disabled the re-engagement of PyDiag in LOW_NOISE_2 and will observe behaviour/performance.

(We tried to post this entry during last night, it appeared on my local session of the logbook, but in fact the post did not work).

The goal of the afternoon shift was to find the longitudinal position of SDB1 (Z) that minimizes the CMRF.

At the beginning of the shift, Maddalena commented the engagement of the BS_TY_SET loop in ITF_LOCK.py.

Then there were several unsuccessful lock trials, two unlocks happened right after reaching CARM_NULL_1F. It was then decided to cancel out the last step performed on the NI DAS around 15h20 utc.

While the ITF was in LOCKED_ARMS_BEAT_DRMI_3F, we saw that the SDB1 bench was moving (the drift control was not working). The galvo loops of B1p QD were also open so we closed them manually at 15h18m42-UTC. After this action, the drift control of SDB1 was back in operation.

After several unsuccessful attempts to lock the ITF finally reached a stable lock in CARM_NULL_1F at 16h37 utc.

ITF locked in DC readout at 16h56m30 utc but we unlocked shortly after at 16h57m06 utc during the transition to Low Noise 1.

DC readout reached again at 17h32m10 utc, but ITF unlocked again at 17h34m14 utc during the transition to Low Noise 1. The ISC team noticed the presence of an oscillation on BS_TX at 275 mHz. For safety they are disabling the full bandwidth control of BS for the next relock.

Noticing that the B1p 56 MHz side bands were very unblanced, an offset was added on MICH to improve the balancing of B1p side bands.

It was then possible to reach DC readout at 18h50m05 utc. and Low Noise 2 at 18h52 utc but ITF unlocked at 18h55m18 utc.

DC readout at 19h57 utc and Low Noise 2 at 19h59 utc. Low Noise 3 reached at 20h26 utc.

The BS_TY_SSFS_LF_Q_ENBL was disabled when arriving in Low Noise 3 (because it had been renabled before for a few min), and the offsets were cancelled to bring BS_TY_INPUT back at 0.

We then moved the SDB1 bench along Z (starting from initial position: SDB1_LC_Z = 3890 um, Sa_OB_F0_X = 2599 um).

We performed steps of 100 um on F0_X, and we adjusted the bench TX offset in parallel by doing steps of -2 urad per 150 s, for each step of 100 um of bench displacement.

We stop at 22h43 utc at SDB1_LC_Z = 960 um, Sa_OB_F0_X = -900 um which seems to correspond to a zero on the B1s signal demodulated at 491 Hz. We take quiet data until 23h06 utc, unfortunately the DCP was only getting worse.

Fig.1 shows the scan of the SDB1 bench longitudinal position and the period of data when we stabilized it at SDB1_LC_Z = 960 um. Data will be analyzed offline.

Fig.2 shows the scan of SDB1 in the opposite direction towards the standard position. The ITF unlocked at 23h49 utc.

After the unlock we restored the bench at its standard position and re-enabled the BS_TY_SSFS_LF_Q loop in ITF_LOCK.

ITF State: LN3
ITF Mode: LOCKED
Quick Summary: ISYS: in standard state, SBE loops closed, Susp loops closed, DSP of BS still offline.
Activities ongoing since this morning: No activities ongoing

The first part of the shift was dedicated to the recovery of the of the ITF up to LOW_NOISE_3 (see report #61755). We reached LOW_NOISE_2 at 14:02 UTC, the lock lasted until 14:21 UTC. Unfortunately after this event we were able to reach LOW_NOISE_3, required for the DET activity, only at 20:25 UTC. 
DET activity concluded at 1:50 UTC due to an unlock, while relocking Gouaty restored the old SDB1 position. ITF left relocking with AUTORELOCK_FAILSAFE engaged. 

DAQ
During the morning two blackout events, happened at around 10:00 UTC and 12:00 UTC, triggered some missing data from various ACS INJ and DET signals. After restarting the processes eurothermServer and BacnetServer and reloading FbsMoni at around 14:10 UTC the signals were achieved again. Also, thanks to the support of Cavalieri and Kraja, the module ENV-WAB was remotely restarted and the signal related to the WAB Building were restored from 15:15 UTC.

ISC
At 0:03 UTC, as agreed with the Commissioning crew, I changed the Etalon WI Setpoint from 20.25 to 20.35 with a ramp of 4 hours (command 'WI:WI Set point [WI_RH_SET=20.35,rampTime=14400]').

Today afternoon, it was not possible to reach LN2 most probably due to the unbalance of the B1p 56 MHz upper and lower sidebands (see fig.1).

In the lock started at 18.11 UTC (when CARM NULL 1F has been achieved), we added -11 as MICH offset in order to overimpose the sidebands  to proceed up to LN2, the trick worked (see fig.2) but the ITF unlocked at LN2 due to huge oscillation at ~55 Hz .  

Thus, to save the weekend, we decided to reverse the last step of DAS WP.

So, at 19.24 UTC, we came back to STEP_1S (61729) .

After that, the ITF achieved LN3 (at 20.25 UTC) without adding extra offsets (fig.3).

As requested by Romain, the BS TY SET was put to 0.

We left the ITF to Romain.

After solving the problem on the BS DSP, DAS tuning started.

9.10 UTC: LN3

9.35 UTC: WI DAS INCREASE (+15 %)

10.01UTC: unlock

10.53 UTC: the sidebands was low in the CITF preventing the lock acquisition→ increase WI CH by 15 %

The powers applied in this step are summarized in the table below.

After that, it was possible to reach CARM NULL 1F but the misbalance of the upper and lower 56 MHz sidebands on B1p prevented the lock in low noise, thus I reverted the increase of the WI DAS power and, following Paolo’s indication, I slightly decreased the NI DAS by 10 % (at 13.28 UTC).

The ITF unlocked during different steps of the lock acquisition, thus, the DAS powers have been re-settled to STEP 5S (at 15.22 UTC).

Sorry for the delay of the start of the afteroon activity. 

Indeed it worked! The bump at 48.1 Hz has completely disappeared.

(the sidebands around the 50 Hz mains line are not related to this action, but developed overnight during the last lock.)

As of mid-August, the 48.1 Hz that we label as "WE crossbar" became more excited. This occurred on Monday, August 14, within an unlock between 11:00 and 14:30 UTC. No particular action is reported in the logbook.

Today at 10:23 UTC we went to WEB and unplugged the mains cables from the motor crates, because we know that in the past there has been an unexpected coupling with some noise flowing through them.
The cables have been temporarily tagged (they are usually connected to a "smart power strip"), so that they can be reconnected in the right location.

We will test whether the 48 Hz noise is reduced at the next lock, after which we will make a final decision for the right system configuration.

It may be worth repeating the same action at NEB as well.

Upon arrival I found ITF LOCKING_IR and SUS+BS Metatron node stuck due to the problem on SUS_BS DSP board (BS gray flags on DMS). It prevented the lock, so I contacted Bersanetti who bypassed the BS top stage in the automation .
  - Experts allerted in case of BS_DSP Board reset or replacement: Passuello, Carbognani, Boschi, Ruggi, Bersanetti
At 08:45 UTC the lock acquisition restarted (COMMISSIONING MODE again set).

The planned TCS/ISC activity on differential DAS tuning (Nardecchia, Spinicelli) started at 09:00 UTC and went on without major problems except few unlocks.  

EnvMoni
12:16 UTC - LargeCoil_CEB process stuck -> stopped / restarted via VPM

Other
INF
two blackout occurred, several INF and ENV apparatus are still offline. Experts informed.

SUSP
(22-09-2023 06:00 - 22-09-2023 08:45) SUS_BS DSP board vs SUSP_BS Metatron node

SUSP
(22-09-2023 06:00 - 22-09-2023 08:00) Operator on site with expert from remote
Status: Ended
Description: SUS_BS DSP board investigation

I modified the code for the suspension nodes in order to skip the checks on the top stage; this applies to BS only. SUS_BS is operational again, there shouldn't be any other places where we try to access BS top stage.

As deemed necessary, the North Arm is being kept locked by putting in pause the nodes SUS_BS, WARM_LOCK and ARMS_LOCK, and acting manually on NARM_LOCK.

SUS_BS DSP board investigation in progress
ITF in TROUBLESHOOTING MODE

ITF State: LOCKING_ARMS_IR
ITF Mode: LOCKING
Quick Summary: ISYS: in standard state, SBE loops closed, Susp loops closed, SUS_BS Metatron node stuck (TBC).
Activities ongoing since this morning: No activities ongoing

The aim of the shift was to check the performance of the FIS injection with the new B1 PDs

List of the activities:

As first action of the shift we measured the Squeezer parametric gain with the HD detector. We had 9mV max and 2.64 mV min  of HD 4 MHz mag. This correspond to a parametric gain of 3.41, i.e. 10.6dB o SQZ produced by the OPA

At 14:55 UTC we started to open the SQB1 Shutter and to rotate the SQB1 HWP2 half waveplate.

At 15:30 UTC with the automation in SQZ_INJECTED we closed the CC and AA loop and we realigned the SQZ beam toward the ITF

At 15:36 UTC we took the shot noise reference

At 15:44:30 UTC we started a phase scan. From the phase scan we measure roughly 6dB of ASQZ and less than 1dB of SQZ. See fig 1.

We found out that the angle of max SQZ level was 1.2 rad and the angle of max ASQZ level was 2.65 rad.

We measured ASQZ at 16:37 UTC and SQZ at 16:44 UTC. In figure 2 you can see in violet ASQZ, in BROWN shot noise taken at the end of the phae scan (16:10 UTC)  and in RED shot noise taken at 15:36 UTC. In green SQZ taken at 16:44 UTC. You can see that the level of the shot noise is different in both the case.

Thus we decided to take longer data stretches of Shot noise and SQZ:

• figure 3: SQZ ingected up to  17 and 30, than shot noise. You can see that at the beginning the noise in hrec was higher and with a lot of fluctuations. Near the closure of the shutter it was better. When we closed the shutter you can see the step in the noise in Hrec. This is compatible with about 1 dB of SQZ. The cause of the noise was probably an high technical noise in the interferometer. It is also interesting that the 4MHz mag decreased by a factor 2%, but it is not enough to explain the high fluctuations at the beginning of the plot. Moreover are evident some correlation between the movement of SDB1 and the level of the  magnitude, and also some correlation between the coupling of the frequency noise in the ITF (SSFS) and the level of measured squeezing
• figure 4: after the shot noise data we injected again squeezing. The level of the noise in hrec is similar to what we found at the end of the SQZ data in the first part of fig 3.
• At 19 we injected ASQZ and we started to check the alignment, to be sure that the change in the B1 4MHz magnitude was not due to some clipping. We did the following acrions 
  • move SQB1 bench in horizontal X to see inf we could improve the level of the 4MHz mag on B1 (no effect)
  • move the Picomotors on M32 and M33 (small improvement)
  • move the retroreflector on SQB1 (small improvement)
  • optimize the SQB1 HWP1 to decrease the power of the spurious beam from the ITF on EQB1. This was done by looking the beat note of the LO and this beam on HD Detector (open LO Shutter, Put fast DAC offset at 20 mV, rotate SQB1 HWP2 to send all toward HD, improve SQB1 HWP1, come back with SQB1 HWP2)
  • At 20:21 UTC all these operation were concluded
• Figure 5 SQZ (0.9dB at 18:45UTC), ASQZ (5.8 dB at 19:15 UTC), Shot (at 18 UTC).
• Figure 6 the same of figure 5 in time domain

Some observations:

• This time we injected 2dB of more SQZ wrt the last shift. We measure 2dB more of ASQZ but the same level of SQZ (maybe less). We had the impression that the technical noises at high frequency are higher than during the last shift
• Figure 7. Comparison of B1 DC and B1 DC_NULL between new and old PDs. In the case of B1_DC null the noise is lower because the dark noise of PDs is lower, but B1_DC has the sane level. This seems due to the fact that the high frequency was not limited only by the dark noise of B1 PDs. Now the distance between B1_DC and B1_DC_NULL is higher wrt the older PDS, See fig 8.
• We checked also the noise in Hrec during the last night, no SQZ inkected. And you can see that the level of hrec had oscillations compatible with an injection of 0.8 dB of SQZ (Fig 9). All these are probably evidence that we are still not shot noise limited.
• When we removed the SQZ the sensitivity dropped of about 1 Mpc. Some effect is visible (fig 10)

We left the system in SQZ_INJECTED and the shutter open on SQB1. We are injecting SQZ all the night (fig11 preliminary data). At 21:15 UTC we closed the SLOW CC Loop (we forgot before) and we increase the CC gain from 20000 to 35000 (we did not update this value in SQZ_MAIN.ini file)

To remove SQZ injection put SQZ_MAIN in ready for FIS injection and close SQB1 shutter: (SQB1 pico, PM34_NC by plus (800000-1000000) steps)

The first part of the shift was dedicated to complete the DAS differential tuning.

At around 15:30 UTC started the planned activity on SGD; activity concluded around 20:20 UTC.

At 20:26 UTC I set the DQSTUDIES mode and then I engaged the autorelock.

Daily and weekly VIM summary plots now display DQSTUDIES segments.

This morning, we proceeded with the DAS differential tuning. In order to try to evaluate the effect of each DAS ring, the NI outer ring has been decreased by 15 %. 

08.17 UTC: Locked at LN2

08.48 UTC: NI Outer ring power change (-15 % wrt the STEP_1S ) 

09.13 UTC- 9.18 UTC: Manuel tested a new filter

09.18 UTC: ITF unlocked

09.25- 09.46 : Diego P. went to the CB on the platform.

At the next relock, the ITF working point appeared degraded (see fig.1).

Thus, in order to investigate the ratio between NI inner and outer ring, the NI Inner ring power has also been decreased by the same amount (-15 %) .

10.40 UTC:  NI Inner ring power change (-15 % wrt the STEP_2S ). 

Also this step had a negative impact on the ITF WP, B1p was very high and we were not able to lock in low noise 2 due to unbalance of B1P PC 56 MHz.

Thus, with the ITF in CARM NULL 1F, the NI DAS inner and outer ring powers have been increased again, at 12.08 UTC.

13.15 UTC: unlocked

13.24:WI DAS powers have been increased by 15%.

At 14.05 UTC, the ITF locked in LN2.

It seems to be a good WP (see fig.2) but it is needed to dedicate some time to confirm it.   

We left these DAS powers identified as STEP 5_S for the afternoon shift. 

The shift was dedicated to the planned TCS/ISC, differential DAS tuning, carried out by Nardecchia and Mantovani. We found the ITF in LOW_NOISE_3 since yesterday night, at 6:00 UTC we attempted to go back to LOW_NOISE_2, but the ITF unlocked during the transition.
During the shift we were able to relock and reach LOW_NOISE_2 multiple times, from 11:05 UTC we set CARM_NULL_1F as target.
Activity still in progress.

DET
During a thermal transient and with the ITF in CARM_NULL_1F, the B1p Galvo loop opened due to saturation of the correction. Loop manually restored at 11:18 UTC and 11:40 UTC.

SBE
SQB2 Horizontal Loop opened at 5:34 UTC. At 6:00 UTC I closed it again via VPM, but the loop opened again at 8:10 UTC. From 8:57 UTC to 10:10 UTC I acted on the motors from 0 to 3 to restore the Horizontal and Vertical position of the bench close to its setpoints. Loop closed again at 10:09 UTC

SUSP
Since around 3:45 UTC of this night the loop signals related to BS ID, Vertical loop and related guardians became unavailable, while the loops remained closed. Thanks to the support of Carbognani we investigated the issue, and found that the board 41033 was not reachable anymore, preventing connection via DSP interface, SuspensionUI and Chassis Client.
Due to the impossibility of Boschi to intervene on the issue we contacted Passuello, who went to the Central Building after an unlock at 9:30 UTC to investigate.
Unfortunately the only action available is to restart the board, which would trigger an opening of BS ID: in case the connection is not restored, it would be impossible to close the loop again without a long maintenance intervention.
In agreement with the experts and the Commissioning Coordinator, we postponed the recovery of the connection.

As for the SDB1 bench, we compared the Hrec structures excited during the SIB1 near-field sweep magnetic injections (#60083) and the noise injections on the SIB1 bench  (#61714).
Below, is a list of matched structures during the two noise injections.

Figure 1: Hrec spectrum during the magnetic (black curve) and mechanical noise injections (TY, TZ  red and yellow curves).

Figure 2, (20-70)Hz: 

• a resonance structure is excited at ~28.1 Hz (TY dof for mechanical injection);
• NE_MIR_permline_ext0 at 37.5 Hz and WE_MIR_permline_ext0 at 56.5 Hz  (TZ dof) are excited;
• broad noise structure in the frequency region ~(60-65)Hz.

Figure 3, (70-140)Hz: 

• a resonance structure is excited at ~76.8 Hz (TY dof);
• a resonance structure is excited at ~101 Hz (TY dof) and 136.15 Hz.

Figure 4: SSFS_LF_line from ASC_pre | SSFS_LF_line from LSC_Acl_Moni @227.1 Hz and its sidebands ±2.09 Hz, ±4.29 Hz (slight excitation of TY dof) and +6.09 Hz are excited.
 

Figure 1 shows the OMC scan performed yesterday.

Figure 2 shows for comparison the last OMC scan I could find, done on Jul 24.

On figure 1 the carier order 2 mode is a bight higher (30mW vs 20mW in July). The order 3-6 modes look the same, modes 7 and 8 are now much brighter by about a factor 2.