The frequency of the NCal and associated PCal lines have been set to about 52 Hz at 19:10 UTC while the ITYF was unlocked.
ITF found in relocking phase, LN3 automatically reachead at 7:05 UTC.
At 7:09 UTC I set Calibration mode to run the requested calibration:
- CALIBRATED_DF_DAILY
- inject_lsc.py
- inject_ssfs.py
- inject_asc_tes.py
At 8:23 UTC ITF in Commissioning mode to run the requested ENV noise injections; the ITF unlocked at 9:25 UTC while performing acoustic noise injections at WE.
The ITF relocked at the first attempt without any manual intervention; Science mode was set at 10:10 UTC but it unlocked again at 10:33 UTC.
After some failed attempts the ITF was again in Science mode at 11:56 UTC.
At 12:24 UTC ITC in Commissioning mode to complete the ENV noise injections (script had been changed by Maria to restart from WE acoustic injectio); the ITF unfurtunately unlocked at 12:29 UTC. At this point it was decided to skip the ENV injections.
The ITF relocked again without any manual intervention and Science mode was set at 13:24 UTC.
ITF left in Science mode with Autoscience ON.
ENV
During the shift there were periods of high wind activity.
The NCals and associated PCal lines have been changed from around 72 Hz to 36 Hz at 6:33 UTC, while the ITF was unlocked.
Comments on two unlocks during ENV injections:
-
the first unlock occurred during the magnetic injection performed in CEB (Figure 1),
-
the second unlock happened during the magnetic injection at WEB, after completing the injections at CEB and NEB (Figure 2).
It was not possible to complete the WEB magnetic injection nor to proceed with the planned acoustic injections.
The output files of CEB and NEB magnetic injections are collected in virgoData/NoiseInjections/MagneticInjectionsO4/output/:
MagneticSweep_NOISE_MAG_CEB-1447786253.txt
MagneticSweep_NOISE_MAG_NEB-1447787183.txt
15:23 UTC ITF in LN3 COMMISSIONING mode
15:40 UTC CALIBRATION MODE (Verkindt), unlock at 15:51 UTC. Ended at 17:37 UTC
17:37 UTC COMMISSIONING mode for ENV noise injections. Two unlocks at 17:59 UTC and 19:17 UTC. Tringali edited ENV_MAIN script to avoid the injections that had been successfully done before these two unlocks.
20:10, 20:44 UTC unlocks while waiting for the sensitivity curve to smoothen in order to try the injections a third time. B4_Cam1 showed irregularities, BS_IP, Seismon, Sea Activity red, troubles achieving LOCKED_ARMS_BEAT_DRMI_1F: injections activity stopped
21:31 UTC ITF left in LN3 SCIENCE mode with AUTOSCIENCE_ON, range oscillating between 40 and 52 Mpc
Comments on two unlocks during ENV injections:
-
the first unlock occurred during the magnetic injection performed in CEB (Figure 1),
-
the second unlock happened during the magnetic injection at WEB, after completing the injections at CEB and NEB (Figure 2).
It was not possible to complete the WEB magnetic injection nor to proceed with the planned acoustic injections.
The output files of CEB and NEB magnetic injections are collected in virgoData/NoiseInjections/MagneticInjectionsO4/output/:
MagneticSweep_NOISE_MAG_CEB-1447786253.txt
MagneticSweep_NOISE_MAG_NEB-1447787183.txt
Today, from 16h30 to 17h45 UTC, I did the following calibration injections:
CheckHrec
Sensitivity measurement
NE, WE and BS optical response measurements, 3 times, separated by 4mn.
The aim of this last measurement is to estimate the variations of the optical responses on a short time scale.
These injections are summarized by the spectrogram plot attached.
Yesterday and today NE, PR, SR, WI mirrors DSP boards and WI marionette boards have been re-synchronized with timing as reported also in Loic's entry. In case of WI a full reboot of the Sc crate was needed. Another consequence of bad synchronization with timing was the glitches on WE F7 (see Ruggi's entry). For this reason both F7 WE boards (LVDT and CD) were rebooted as well, apparently fixing the issue (see plot). We profited from the above interventions in order to re-center both WE vertical accelerometers that were generating an high DC on the corrections.
Yesterday afternoon and today, Valerio and the SAT team have reloaded and/or rebooted the different suspension DSPs to re-synchronize them.
As shown in the attached figures, the monitoring channels are back to their nominal values for all the suspensions (including SR). Note that for online dataDisplay, the channels are available only late in the data collection pipeline, i.e. in the FbmStDy stream, but not in the other online streams.
For NE and WE marionettes, there is no monitoring and the alert shelving in the DMS ended this week. I have shelved again the 6 associated flags(until January 12th 13h30).
ITF found in LOW_NOISE_3 and SCIENCE Mode with AUTOSCIENCE on.
All times are UTC.
Relevant events during the night (ITF with Autoscience on):
- 00:45:11 unlock from LOW_NOISE_3;
- 01:11:25 unlock from LOCKING_OMC_DARM_B1_PD3;
- 01:23:27 unlock from CARM_NULL_3F;
- 02:07:47 reached LOW_NOISE_3;
- 02:09:08 unlock from LOW_NOISE_3;
- 04:57:45 START LOCK ACQUISITION (ITF NOT_LOCKED (0d 2h 48m 37s));
- 04:57:48 and 05:04:00 unlocks from FMODERR_TUNED;
- 05:11:08 unlock from ALIGNING_DRMI;
- 05:55:37 SCIENCE Mode;
At 10:37:27 ITF unlocked, COMMISSIONING Mode set with LOCKED_ARMS_BEAT_DRMI_1F requested by Was to investigate ~15 minutes glitches.
From 10:57 to 11:47 WI mar, SR mir and PR mir DSP resyncronization, ITF put DOWN by the operator (Boschi, Pierdibene, entry???).
At 11:54:02 ITF in LOCKED_ARMS_BEAT_DRMI_1F.
From 13:03 to 14:00 Boschi and Pierdibene went to West End building for work on WE F7 control signals issues related to ~15 minutes glitches highlighted by Ruggi in #68213.
At 13:17 SBE_SWEB_loop_act_hor_10Hz opened, closed at 14:02.
From 14:05 SCAN_ARMS_ALS requested by De Rossi for ALS alignment improve.
From 14:10 Boschi and Pierdibene working on WI DSP boards rebooting SUSP rack.
From 14:32 Continue with lock acquisition after cross alignment.
ITF left in CARM_NULL_1F and in COMMISSIONING Mode.
I' ve made a simple analysis (too simple to be bullet proof since it doesn t take into account the loop). I ve taken since the start of the run all the data set for which the ITF was locked in steps >carm 0 for at least 2h and unlocked for 2h later and compared the delta in temperature from the unlock (when the YAG just disappears) and 1h later.
It is visible that the response of the WI has been always higher. There is an increment of the delta (higher delta means higher absorption) in the last 30 days, but it is not proven to be a stable behavior and a better analysis has to be performed in order to take into account the loop.
There are coincident glitches in the WE F7 control signals. They started on Monday, after the timing crisis. The involved signals are the LVDT's received from the coil driver board and reconstructed as error signals of the loops. The raw signals, sent to the master board from the LVDT board, look fine, while the recontructed signals are corrupted, so we guess that the issue is at the level of the coil driver, or in the communication between LVDT and CD.
Figure 1. The problem can be already seen in CARM NULL 1F, in the trend data DARM has also glitches spaced by 15min before we reach LN3 (before there is power on B1)
Figure 2. Looking at raw data in CARM NULL 1F, the issue is actually more clearly visible in PRCL, in trend data the high frequency fluctuations of DARM are supressed, which is the reason in trend data it is more clearly visible in DARM.
Figure 3. In RDS 50Hz data the glitches can be clearly seen in PRCL, both in CARM NULL and in LN3
Figure 4. The problem was already present in the night from Nov 18 to 19, when we had the first longer CARM NULL lock after the electrical incident of Nov 17.
Figure 5. The problem was not presen in the morning of Nov 17 before the electrical incident.
A question is if the glitches remain synchronized to the ~15min spacing between different locks.
Another question if is there is any suspension glitches or environmental glitches that are coincident. By glitch, I mean a transient at ~1.3Hz, which means it will not be detected by omicron which typically looks only at higher frequencies.
I performed some quick analyses of these glitches and report a few preliminary observations below.
- Although they appear very strong in B1, their SNR in the strain channel is about 220-270, roughly half that of the 25-minute glitches. Figure 1 shows the glitchgram of the strain data, where these glitches have a central frequency around 100 Hz.
- Their waveform differs from that of the 25-minute glitches: they have a broader bandwidth and are shorter in duration, although they do show residual excitations after the peak: Figure 2.
- They seem to have a preferred separation of roughly 14′45″ ± 2″ (to be confirmed with more statistics), but they can also occur very close to one another: Figure 3.
- I ran UPV on both the strain and on SDB2_B1_PD1_D, but I did not find any evidence of correlations for these glitches (some insights for others). I will repeat the analyses over the next few days with larger statistics.
ITF found locked at LOW_NOISE_3 in COMMISSIONING mode; BNS Range ~49 Mpc. Unlocks (times in UTC):
14:23 - from LN3 due to BS glitch (pdf 1)
14:34 - from REDUCING_CARM_OFFSET_3_OF_3
14:45 - from LOCKED_ARMS_BEAT_DRMI_3F, causing EIB loops to open. Closed at 14:50 by Menzione.
17:19 - from LN3 just after strong light injections (BNS Range <10 Mpc) finished and ITF began returning to its normal working point (pdf 2). Taking advantage of downitme I closed CEB door (fig.1)
18:49 - from LN3 due to BS glitch(? pdf 3)
Relock and commissioning activity (times in UTC):
14:30 - De Rossi entered IER to restore IMC PDH loop UGF to high enough value (entry #68203)
15:10 - 15:42 - Boschi entered CEB and reset DSP timing for WI_MAR and NE_MIR, then left the door open (on purpose? is he new here?).
16:15 - Recovered SQB2 and FCIM SBE vertical position.
16:35 - Relocked to LN3 without changing MICH_SET or misaligning SR.
16:37 - 17:18 - Scattered light injections by Van Haevermaet.
18:12 - Relocked to LN3 on first attempt (MICH_SET=40, SR_TY+=2 urad). BNS Range ~51 Mpc
18:30 - Added MICH offset ramp in automation at CARM_NULL_1F (lines 4584 - 4585 - in ITF_LOCK.py). SCIENCE mode activated via Autoscience at 18:35.
19:32 - ITF auto-relocked to LN3 on first attempt, back in SCIENCE (Autoscience ON) at 19:36. BNS Range unstable, oscillating between 45-50 Mpc; huge glitches like the one observed at at 21:48 caused it to drop down to 30 Mpc.
ITF left locked.
While looking at lock acquisition I have noticed today a new family of glitches that have not been happening before Monday.
Figure 1 shows a few of those glitches today, they are extremely loud so they are visible directly in the B1 DC time series. They cause saturation of the B1 photodiodes and often kicks in the SDB1 alignment that are caused by these B1 photodiode saturations. The spacing looks very regular, with one glitch every 15 minutes.
Figure 2 these glitches happen in multiple locks, and might be causing lock losses.
Figure 3 looking at the timing of the lock losses, it appears roughly at the time of when the next 15 minute glitch should occur
Figure 4 these glitches have a shape of an oscilation at ~1.3Hz
These glitches will be a problem to keep the interferometer locked. I wonder if they are in addition to the 25 minute glitches, or if the 25 minute glitches have been replaced by a much louder and more frequent version.
I performed some quick analyses of these glitches and report a few preliminary observations below.
- Although they appear very strong in B1, their SNR in the strain channel is about 220-270, roughly half that of the 25-minute glitches. Figure 1 shows the glitchgram of the strain data, where these glitches have a central frequency around 100 Hz.
- Their waveform differs from that of the 25-minute glitches: they have a broader bandwidth and are shorter in duration, although they do show residual excitations after the peak: Figure 2.
- They seem to have a preferred separation of roughly 14′45″ ± 2″ (to be confirmed with more statistics), but they can also occur very close to one another: Figure 3.
- I ran UPV on both the strain and on SDB2_B1_PD1_D, but I did not find any evidence of correlations for these glitches (some insights for others). I will repeat the analyses over the next few days with larger statistics.
Figure 1. The problem can be already seen in CARM NULL 1F, in the trend data DARM has also glitches spaced by 15min before we reach LN3 (before there is power on B1)
Figure 2. Looking at raw data in CARM NULL 1F, the issue is actually more clearly visible in PRCL, in trend data the high frequency fluctuations of DARM are supressed, which is the reason in trend data it is more clearly visible in DARM.
Figure 3. In RDS 50Hz data the glitches can be clearly seen in PRCL, both in CARM NULL and in LN3
Figure 4. The problem was already present in the night from Nov 18 to 19, when we had the first longer CARM NULL lock after the electrical incident of Nov 17.
Figure 5. The problem was not presen in the morning of Nov 17 before the electrical incident.
A question is if the glitches remain synchronized to the ~15min spacing between different locks.
Another question if is there is any suspension glitches or environmental glitches that are coincident. By glitch, I mean a transient at ~1.3Hz, which means it will not be detected by omicron which typically looks only at higher frequencies.
There are coincident glitches in the WE F7 control signals. They started on Monday, after the timing crisis. The involved signals are the LVDT's received from the coil driver board and reconstructed as error signals of the loops. The raw signals, sent to the master board from the LVDT board, look fine, while the recontructed signals are corrupted, so we guess that the issue is at the level of the coil driver, or in the communication between LVDT and CD.
The result of the NE survey is reported in the entry 68206
The result of the WE survey is reported in the entry 68205
Today we carried out the survey for measuring the position of the NCal benches @NE: as for the previous survey, we measured the point on top of the calibrator and the points located at the sides of the benches.
After the first survey of the two benches (east and west side of the tower) the calibrators have been swapped each other and the survey repeated. The second position is labelled with the addition of _Swap.
Tab1 reports the result of the survey, while the scheme of the survey is shown in the figure NE_NCal_SurDiag.
Fig.1 to 5 report some pictures of the survey carried out and of the target points.
Yesterday, we carried out the survey for measuring the position of the NCal benches @WE: as for the previous survey, we measured the point on top of the calibrator, 2 for each bench and the points located at the sides of the benches (where possible)
After the first measure on the top, each calibrator has been turned by 180 degrees and the survey repeated. The second position is labelled with the addition of _Pos2.
Tab1_1 and Tab1_2 report the result of the survey, while the scheme of the survey is shown in the figure WE_NCal_SurDiag.
Fig.1 to 5 report some pictures of the survey carried out and of the target points.
# 2025-11-20 1441046441
SCATL z 1447691935 300 WE MIR # SWEB freq=0.1 amp=140
SCATL z 1447692266 300 NE MIR # SNEB freq=0.1 amp=140
SCATL z 1447692603 300 WE MIR # SWEB freq=0.2 amp=140
SCATL z 1447692933 300 NE MIR # SNEB freq=0.2 amp=140
SCATL z 1447693269 300 WE MIR # SWEB freq=0.3 amp=140
SCATL z 1447693599 300 NE MIR # SNEB freq=0.3 amp=140
SCATL z 1447693936 150 WE MIR # SWEB freq=0.4 amp=80
SCATL z 1447694117 150 NE MIR # SNEB freq=0.4 amp=80
The last one was a test at 0.4Hz with lower amplitude to see how it would compare with lower-higher freq-ampl, and both SWEB and SNEB showed expected LC_Z movement, at SNEB we could even see the second order scattering now. Unfortunately the amplitude was probably still to high and right after the end of the injections the ITF unlocked due to the last injections.
Maybe next time such a test could be done in LN2.
Since yesterday evening some instabilities of the INJ system have been observed (see as ex. plot 1: just before the IMC unlock the EOM corrections increased).
We measured the IMC TF, finding a very low UGF (around 75kHz, fig. 2).
The only modification on the system we made yesterday has been the swap of the NEOVAN chiller (#68190), which reduced the temperature of the neovan and slightly modified the coupling in the IMC (see IMC REFL decrease, plot 3). Not sure if it is the only cause of this change in the loop behaviour. Moreover, the temperature seems less stable (tomorrow we will decide whether to swap again the chiller, which is now working again thanks to fabio and cecilia).
By decreasing the attenuation of the IMC PDH loop from 21dB to 16dB (i.e. increase 5dB the gain) we recovered a more usual UGF and phase margin (fig.4)
ITF found in TROUBLESSHOOTING mode and in LOCKING_ARMS_IR but with Injection system down since 01:00 UTC due to a failure of Neovan electronics. OnCall expert recovered the standard state at 07:45 UTC.
07:48 UTC: COMMISSIONING mode set. After the usual cross-alignment in ACQUIRE_DRMI, under request of Mantovani, I moved SR about 2 urad tx+ at LOCKED_ARMS_DRMI_1F and waited 1 minute in LOCKED_ARMS_DRMI_3F. After several attempts, ITF has been relocked at CARM_NULL_1F in a stable way.
LN3 acheived at 09:26 UTC. Mantovani and Boldrini revovered a good ITF working point.
ITF locked at ARMS_IR during the activity on NCal at NE. Task conluded at 11:18 UTC. ITF relocked at CARM_NULL_1F in a stable way. Back at LN3 at 12:57 UTC.
Planned activities "probably in progress" but not communicated to the Control Room:
- ENEL external electricity interuption started at 07:30 UTC (D'Andrea)
- FC air conditioning concrete extension (Fabozzi)
- CEB/NEB/WEB/MCB door replacement (Fabozzi)
- Soil seismic study, arms (Paolucci)
Planned activity communicated to the Control Room:
- Ncal rotor position metrology (Mours, INF) - Concluded at 11:18 UTC.
ISYS
(20-11-2025 07:00 - 20-11-2025 07:45) On site
Status: Ended
Description: ISYS system DOWN, failure of Neovan electronics. Recovered by OnCall expert.
Following the recovery of this morning, there are alerts in the DMS (triggered only in LN3 mde): some DSPs from different suspensions are not properly synchronized:
- the NE, WI and PR MIRrors have a different synchronisation that before the timing issue of last Monday. In particular the case of the NE mirror will introduce large errors in the reconstructed h(t) data.
- the synchronisaton of the NE and WE MARionnettes is not activated since few months, so no information on them. But since they are not used for the controls, it has no impact on Hrec.
The first two VIM plots from today show the current status for the mirrors and for the marionettes (the phase is computed at a line injected at 2012 Hz):
- NE Mirror DSP off by 50 µs --> impact on Hrec
- WI Mirror DSP off by 50 µs
- PR Mirror DSP off by 50 µs
- SR MIrror DSP off by 25 µs
- WI Marionette DPS off by 25 µs --> impact on Hrec
The figures 3 and 4 show the monitoring data on Monday 17 November: the synchronisation of the different DSPs changed at different times in the afternoon.
The last figure shows the current bias on hrec (supposed to be unbiased) : this has a huge impact on the phase bias, (apparently not on the h(t) amplitude, which I do not undertand at first sight). The phase bias looks consistent with ~50 µs bias around 250 Hz.
At least the NE mirror and WI marionette DSPs should be resynchronized as soon as possible to have a correct reconstructed h(t).
Yesterday afternoon and today, Valerio and the SAT team have reloaded and/or rebooted the different suspension DSPs to re-synchronize them.
As shown in the attached figures, the monitoring channels are back to their nominal values for all the suspensions (including SR). Note that for online dataDisplay, the channels are available only late in the data collection pipeline, i.e. in the FbmStDy stream, but not in the other online streams.
For NE and WE marionettes, there is no monitoring and the alert shelving in the DMS ended this week. I have shelved again the 6 associated flags(until January 12th 13h30).
20/11/2025
This morning, we performed a geometrical survey of NEB NCals and swapped NEN and NWN rotors. Here below the details.
All times in UTC. Use notation as in wiki page
7:13 : Jigs placed on NE and NW platines between 7:13 and 7:14 (see position plot)
Positions of rotors before survey and changes:
-NWN : rotorbox 2021-06 (wiki table says this box contains R4-17)
-NEN : rotorbox 2022-01 (wiki table says this box contains R4-16)
~8:53 Start calibrating station 1 for measurement of East setup
~9:19:56 : NEF measure; --> fail?
remeasure calibrating points from 9:26:52 to 9:34:28
9:34:28 : NEF point
9:35:05 : NEN point ->9:35:49
9:36:34 : NET point --> 9:37: 16
9:37:50: point 210_aux (to check between stations)
9:42:33 : Start dismounting NCal NEN
9:56:17 Start calibrating station 2 for measurement of West setup
10:00:07 sphere on NWF point
10:00:38 sphere on NWN point
10:01:34-49 move sphere back to NWF
10:02:35: move sphere to NWT --> 10:03:24
measure 210-aux and put rotorbox 2022-01 on NWN
10:08:35 measure NWN_swap; 10:09:22 measure NWF_swap - 10:10:05 measure NWT_swap -->
10:16:29 rotorbox installed on rotorbox 2021-06 on WEN
~10:23:01 start calibrate station 3 for East "swap" measurement
10:25:38 NEF -> 26:13-26:21 NEN -> 26:46-27:14 NET ->27:45-28:04 210_aus 28:54
Remove jigs from East setup (10:29:39-10:30-15)
WEB_pos_sensors.png shows the position measurement of East and West setup during all the today work
11:00 unexpected LV problem for NNN observed: 32V gives 0 current.
- change it from channel 1 ncal east Power Ssupply to channel 2 ncal south Power Supply
- It did not fix the problem unfortunately
11:05 we restart all rotors (except NNN which is not spinning) at twice the nominal frequency and leave
The result of the NE survey is reported in the entry 68206
Earlier this week there was also that behavior. A MICH offset by itself doesn't help. When the is an SR TY misalignment of 2 urad, then the sideband get misbalanced, and the MICH offset restores the sideband balancing. With SR TY misaligned, without a MICH offset, when the dark fringe offset for locking the OMC is put it the power on B1p increases by a factor ~3 and becomes too high. Adding a MICH offset when SR TY is misaligned removes that problem. So at the moment SR TY misalignment and MICH offset are a combined package where both need to be put in at the same time to help, one of them on its own doesn't help, and actually makes some things significantly worse.
The SR TX misalignment doesn't seem to have that issue of affecting the sideband balancing. So maybe a MICH offset is not needed in that case, and just reducing temporarily the SR TX loop gain between CARM NULL 1F and LN3 would be enough to make that idea of SR TX misalignment work in a systematic way.
Note that the sideband balancing changes when staying in CARM NULL 1F for one hour, so what may be true for 15 minutes after reaching CARM NULL 1F, may be no longer true one hour after reaching CARM NULL 1F.