DRAFT - REPORT IN PROGRESS - will be finalized at the end of the shift
ITF found in LOW_NOISE_3 and in Science Mode.
It unlocked at 15:46 UTC, back in Science Mode back in 16:39 UTC.
I was able to close the WI ID loop without issues by applying the standard procedure. After that, I removed the Earthquake mode and realigned the cavities. Then I started the lock acquisition, but there was a problem with the north cavity: although it initially locked, the lock was lost after a short time.
At 05:27 UTC, I set Troubleshooting mode and began contacting the ISC on-call and the SUSP on-call. Around 07:00 UTC, I also contacted Paolo.
After some investigation, he discovered an incorrect value for the gname SAT_NI_MAR_ENBL. Once this issue was fixed, we were able to relock the cavities; unfurtunately the ITF keep unlocking in CARM_NULL_3F. The ISC on-call was still connected and continued investigating the problem.
We discovered that this gname can be modified by clicking the "Reallocation" button on the SUSP GUI. We also found that two other gnames were affected: SAT_NE_LOWN_ENABL and SAT_WE_LOWN_ENABL.
In the end, we were able to restore the correct values, and finally, at 10:50 UTC, the ITF was put into Science Mode.
Guard tours (time in UTC)
11:30-12:00
ITF found in LOW_NOISE_3 and SCIENCE mode (Autoscience ON); BNS Range ~53 Mpc.
ITF unlocked at 01:36UTC due to end TM (MIR or MAR) correction saturation (pdf 1). WI Guardian opened F0 loops, reflections caused by movement of WI visible on WE_CAM. ITF set to DOWN in EARTHQUAKE mode waiting for WI oscillatons to subside.
Attempted to close WI F0_DC_ENBL at 02:50UTC with no success. Second attempt at 04:02UTC unsuccesful again, ISC on call notified.
Recovered at 05:22UTC thanks to intervention by Berni, lock acquisition in progress.
Guard tour (UTC)
-> 21:38
23:02 -> 23:45
01:05 -> 01:55
03:02 -> 03:42
ISC
ViolinModes red flag on DMS excited at around 00:50 UTC (fig.1)
When I arrived, locking acquisition was in progress
- 13:31UTC, Science mode
- 19:30UTC, Calibration mode to perform the planned measurements
- 19:31UTC, Check Hrec
- 20:02UTC, ISC injections
- 20:21UTC, back to Science
Guard Tours (UTC):
12:47 - 13:20
15:47 - 16:24
18:12 - 18:47
20:57 -
Today (Jul 5) around 18:00 UTC I have set back the ASC_SR_TX_SET to 0.022 with a 2 hours long ramp. As it was reset a few hours before by the unlock, and the BNS range has been lower by a few Mpc since that relock.
ITF found in LOW_NOISE_3 and in Science Mode.
At 10:53 UTC there was a temporary lack of channels, while investigating I've found this message on SUSP_Fb: 2025-07-05 10h53m53 UTC frame 1435748051 not send to FbmFFE: sending queue is full.
The ITF unlocked at 12:21 UTC and, after a period of instability of the Injection, it was possible to start relocking at 12:42 UTC.
Relock in progress.
Guard tour (UTC)
04:56 - 5:32
10:12 - 12:40
12:47 -
ITF remained locked in LOW_NOISE_3 with SCIENCE mode (Autoscience ON) for the whole shift; BNS Range ~54 Mpc.
Guard tour (UTC)
22:35 -> 23:08
00:30 -> 01:08
03:05 -> 03:40
04:56 -
- guard tours UTC):
20:00 --> 20:40
I extended the analysis of the BNS range fluctuations over the past two days to include all _mean and _rms channels from the trend frames (~15k in total). The most correlated channels are those related to B5, i.e., the beam at the dark port before the Output Mode Cleaner. This is probably not informative.
Other notable channels include: V1:Sc_BS_CMRF, the control of the Common Mode Rejection Factor at the Beam Splitter, and V1:Sa_WI_F0_ACC_X, the X-axis acceleration at Filter 0 of the West Input.
Attached:
- The text file with the full ranking of channels showing a Pearson correlation greater than 0.2 (in absolute value),
-
and a few plots of the most correlated channels.
Following the analysis done in the past days, which showed correlation between B1p DC (and the B1p beam shape) and the BNS range, we decided to make a scan of the SR tx in order to try to drive this effect.
In the Figure 1 a trend of the test is visible and we can say that for SR tx offset of ~ 0.022
- the B1p DC is minimum
- the OG is lower than the 0 position
- the frequency noise coupling is higher than the 0 position
but the BNS range is maximum; this may lead to a lower mistery noise? to be understood
We leave for this lock a ramp going to the optimal point for SR tx (which will be reset at the next unlock)
Today (Jul 5) around 18:00 UTC I have set back the ASC_SR_TX_SET to 0.022 with a 2 hours long ramp. As it was reset a few hours before by the unlock, and the BNS range has been lower by a few Mpc since that relock.
I analyzed the LSC noise injections done last Monday, 30/06. In the plots the results on DARM and Hrec.
There is basically no difference with the recent past: CARM is still affecting DARM (ISC recalibration still pending) but not Hrec because of CAL own measurements; Hrec is instead affected more by MICH up to 30 Hz.
Figure 1During the step of the ring heater on June 20 the modes order 7 and 11 have moved in opposite direction, while simple simulations of a single arm would expect for all modes to move in the same direction in the arm FSR, with the change in the FSR smaller for very high order modes as they start to be clipped in the arm.
I have reused the Finesse 3 simulation that Augustin has been developping for scattered light (https://git.ligo.org/augustin.demagny/finesse-simulation-04) to look instead at the CARM to B1 transfer function. These simulations include a misalignment of SR by 2urad. I have added appertures of 170mm in radius to add realistic losses to very high order modes, and simulated with the max TEM number of 10.
Figure 2 shows the result around the first FSR for a nominal end mirror radius of curvature of 1683 meters, while Figure 3 shows the same for 1679m radius of curvature for both end mirrors. It shows the same effect that the two HOM spanning around the first FSR of the arm move in opposite directions, which is counterintuitive, but similar to what has been observed experimentally on figure 1. Note that peak in the middle of the figure is slightly above 50kHz, while the analytical calculation predicts an FSR with a spacing of 49'969 Hz (slightly below 50kHz). Does this mean that the peak we see is not the FSR, but something else? What is strange is that it remains in the simulation when the max TEM is reduced to 0, so it cannot be the order 9 mode. The side modes around 44kHz and 56kHz, also remain present for max TEM 1, and disappear for max TEM 0, so this is actually two images of the order 1 mode, and then it makes sense that they move in opposite direction when the RoC is changed.
Figure 4 and 5 shows the frequency range of modes order 1-3, respectively for 1683m on EM and 1679m. These modes behave as expected, moving to lower frequency for a shorter radius of curvature.
Figure 6 and 7 shows the frequency range of modes 4-6, the picture is confused with more modes than one would expect. Some of the modes move to higher frequency for a shorter RoC, while others move to lower frequency. In particular the mode around 33kHz, which could be the order 6 mode moves to higher frequency, so the opposite of the low order modes.
I will add the modified code into git under 'high order mode - CARM.ipynb'.
I’m just adding a spectrogram to better highlight the data segment and frequency range affected by this noise issue.
ITF in Science for the whole shift, BNS Range ~53Mpc
VPM
08:35UTC, SNEB_Quadrants process stopped for a mistake, restarted it properly after 1 minutes
Last night, there was a malfunction of the control of one of the NCal that did not correctly measure the rotation speed. This was triggered by a drop of the LED power around 20:45 UTC. This ended this morning slightly before 3:00 UTC. As a consequence, two new lines were present during this period, one at exactly 36.36 Hz, the other one, not very stable, around 39 Hz.
There was another short occurrence of this issue around 8:30 UTC.
To mitigate this issue, we plan to adjust the NCal LED threshold as soon as possible.
I’m just adding a spectrogram to better highlight the data segment and frequency range affected by this noise issue.
Figure 1 shows the spectrum for the 56MHz TEM00, both USB and LSB, normalized to be units of RIN, and with the contribution of shot noise and PD electronic noise quadratically subtracted. The spectrum looks the same for both sidebands, and there was no significant change during the 2.5 hours separating the repeated measurement of the USB, which is a sign that the alignment has not drifted in the meantime, the beam jitter peaks are small in all cases. The requirement (VIR-1225B-19) for the RAM servo is 2.5e-8 V/V/rtHz at 100Hz, which should correspond to RIN of the sideband of 5e-8 1/rtHz. So probably RAM is a significant part of that spectrum, but this would need to be checked further, as the RAM servo was better than the requirements from what I remember.
Figure 2 shows the coherence with B1s, it is substantial at ~30%. It means that the sidebands are a major contributor to the dark fringe fluctuations in the 100Hz and 500Hz. The coherence for the red trace is smaller as there is a loud glitch on B1s dominating the spectrum during that measurement.
There is no broadband coherence of the sideband with h(t).
Figure 3 shows the series of measurements with the first measurement being the 56MHz USB, and the last measurement being also the 56MHz USB
Figure 4 shows the spectrum of the higher order modes from 2 to 10, with two measurements for the mode order 9 (vertical and horizontal peaks). The PD electronic noise is 1e-8 mW/rtHz, and for the peaks with low power, the spectrum at 1kHz is very close to that limit. There is about a factor 10 difference in power between the brightest mode (order 3) and the dimmest (order 10). The B1 photodiodes have a lower dynamic (~5mW), which would be sufficient as the brightest mode has 1mW in this scan, and have a lower electronic noise at 6.5e-9, so that could be a simple solution to gain a factor sqrt(2) in SNR for these measurements.
Figure 5 shows the same normalized in units of RIN and with the shot noise and electronic noise quadratically subtracted. In these units the noise level is more similar between the modes, the beam jitter peak height varies by one order of magnitude between different modes. There could be a similar level (within a factor 2) of noise that has a power-law shape around 100Hz, with a slope between 1/f and 1/f^{2/3}. That level of noise is a factor 2 higher than what is measured with the 56MHz sideband, so it is not intrinsinc to the measurement process, but HOM may be more affected than a TEM00 mode by beam jitter.
Figure 6 shows that none of those mode is coherent with B1s down to 0.1% level, which is surprising. It tells us that the EDB OMC is adding new information that is not accessible using only B1s, but it is unclear what that information means. There is coherence in the 20-30Hz band where the SRCL and MICH lines are in LN2 and at ~156Hz where the mechanical mode of SIB1 is.
Figure 7 shows the coherence with h(t), there is no broadband coherence to the 0.1% measurement noise level.
Figure 8 and 9 shows mode 3 and 5 which have the powerlaw slope the most clear, as they have high power and low amplitude for the beam jitter peaks
Figure 10 and 11 shows mode 2 and 4 which have a significant lower noise than modes 3 and 5
/users/mwas/OMC/RINanalysis_20250703/
ITF found locked at LN3 in SCIENCE mode. It kept the lock for the whole shift.
ITF found in LOW_NOISE_3 and SCIENCE mode (Autoscience ON); BNS Range ~54 Mpc. Manually unlocked at 15:06UTC for planned EDB OMC HOM noise investigation at LN2 in COMMISSIONING mode.
ITF achieved LOW_NOISE_2 at 15:44UTC. EDB Picomotor process found dead on VPM (connection to picomt13 failed). Fixed by manually restarting the controller in DET lab and the EDB_PICO process on VPM at 16:36UTC thanks to timely, pivotal intervention by Spinicelli.
ITF back to LN3 in SCIENCE at 19:34UTC after end of HOM study.
Guard tour (UTC)
17:57 -> 18:35
20:53 ->
EnvMoni
Lights in MCB_TUNNEL and CEB_HALL show yellow warning on DMS (fig 1). During intervention on EDB picomotors lights could be seen underneath the hanging plastic screens around SA towers in central building.
Other
17:49UTC TelescreenLeft on VPM reports critical alert: wrong or no stream declaration for '1500WCleanRoom1'
Today I have locked the EDB OMC again on HOM in LN2. This time without realigning the OMC after the first initial realignment. There is no coherence with any of the HOM that I have tried.
Initially the EDB picomotor process could not be reached. I must have forgotten to switch off the server last week, and it lost connection on Sunday. Piernicola and Bernardo turned on and off the picomotor manualy, which restarted the connection.
aligned EDB OMC on 56MHz USB, first to maximize the power, then ~50
picomotorse in horizontal to reduce the jitter peaks
16:47 UTC (5min) - locked on 56MHz USB
when going through carrier TEM00 there is ~0.1 mW on it. The 6MHz TEM00 are much smaller, around 0.02mW.
Adjusted demodulation phase to put all the signal in the i quadrature, this must have increased the gain by factor 3 or 4 (and the calibration)
17:09 UTC (5min) - locked on carrier TEM00 (very noisy and low power), not sure if the locked actualy worked or just stabilized in temperature around the mode
Reduced the gain by factor 2 (and calibration of error signal, still
probably wrong by factor 2) as the loop was oscillating when locking
on TEM00 56MHz LSB
17:19 UTC (5min) - locked on 56MHz LSB
order 1 mode of 56MHz sideband has about 0.05mW of power, 10 times less than the TEM00
17:30 UTC (5min) locked on carrier order 2 mode, mostly horizontal
17:40 UTC (5min) locked on carrier order 3 mode, 1-vertical, 2-horizontal
17:50 UTC (5min) locked on carrier order 4 mode, 4-vertical, 0-horizontal
17:59 UTC (5min) locked on carrier order 5 mode, 0-vertical, 5-horizontal
18:07 UTC (5min) locked on carrier order 6 mode, 1-vertical, 5-horizontal
18:16 UTC (5min) locked on carrier order 7 mode, 1-vertical, 6-horizontal
18:26 UTC (5min) locked on carrier order 8 mode, 1 vertical, 7 horizontal
18:35 UTC (5min) locked on carrier order 9 mode, 9 vertical, 0 horizontal
18:46 UTC (5min) locked on carrier order 9 mode, 1 vertical, 8 horizontal
18:56 UTC (5min) locked on carrier order 10 mode, 0 vertical, 10 horizontal
19:06 UTC (5min) locked on 56MHz USB TEM00, alignment is still good
Figure 1 shows the spectrum for the 56MHz TEM00, both USB and LSB, normalized to be units of RIN, and with the contribution of shot noise and PD electronic noise quadratically subtracted. The spectrum looks the same for both sidebands, and there was no significant change during the 2.5 hours separating the repeated measurement of the USB, which is a sign that the alignment has not drifted in the meantime, the beam jitter peaks are small in all cases. The requirement (VIR-1225B-19) for the RAM servo is 2.5e-8 V/V/rtHz at 100Hz, which should correspond to RIN of the sideband of 5e-8 1/rtHz. So probably RAM is a significant part of that spectrum, but this would need to be checked further, as the RAM servo was better than the requirements from what I remember.
Figure 2 shows the coherence with B1s, it is substantial at ~30%. It means that the sidebands are a major contributor to the dark fringe fluctuations in the 100Hz and 500Hz. The coherence for the red trace is smaller as there is a loud glitch on B1s dominating the spectrum during that measurement.
There is no broadband coherence of the sideband with h(t).
Figure 3 shows the series of measurements with the first measurement being the 56MHz USB, and the last measurement being also the 56MHz USB
Figure 4 shows the spectrum of the higher order modes from 2 to 10, with two measurements for the mode order 9 (vertical and horizontal peaks). The PD electronic noise is 1e-8 mW/rtHz, and for the peaks with low power, the spectrum at 1kHz is very close to that limit. There is about a factor 10 difference in power between the brightest mode (order 3) and the dimmest (order 10). The B1 photodiodes have a lower dynamic (~5mW), which would be sufficient as the brightest mode has 1mW in this scan, and have a lower electronic noise at 6.5e-9, so that could be a simple solution to gain a factor sqrt(2) in SNR for these measurements.
Figure 5 shows the same normalized in units of RIN and with the shot noise and electronic noise quadratically subtracted. In these units the noise level is more similar between the modes, the beam jitter peak height varies by one order of magnitude between different modes. There could be a similar level (within a factor 2) of noise that has a power-law shape around 100Hz, with a slope between 1/f and 1/f^{2/3}. That level of noise is a factor 2 higher than what is measured with the 56MHz sideband, so it is not intrinsinc to the measurement process, but HOM may be more affected than a TEM00 mode by beam jitter.
Figure 6 shows that none of those mode is coherent with B1s down to 0.1% level, which is surprising. It tells us that the EDB OMC is adding new information that is not accessible using only B1s, but it is unclear what that information means. There is coherence in the 20-30Hz band where the SRCL and MICH lines are in LN2 and at ~156Hz where the mechanical mode of SIB1 is.
Figure 7 shows the coherence with h(t), there is no broadband coherence to the 0.1% measurement noise level.
Figure 8 and 9 shows mode 3 and 5 which have the powerlaw slope the most clear, as they have high power and low amplitude for the beam jitter peaks
Figure 10 and 11 shows mode 2 and 4 which have a significant lower noise than modes 3 and 5
/users/mwas/OMC/RINanalysis_20250703/
In the last few days we had (again) three occurrences of a rather obscure problem, where the management of ITF_LOCK by ITF_CONDITIONS (Autorelock & Autoscience node) conflicts with some leftover request from the last management from CALI, in these cases the last requested state during Monday's calibration shift, LOCKED_PRWI.
These requests happen after an unlock, as then is when we request again something to the node, and they are not the cause of the unlock, as stated.
Looking at the unlock from this morning, the SDB1 fast shutter closes for some reason at 06:10:09.93 UTC; the signals based on SDB* are immediately cut off, while the other ones are alive for a bit more time. Everything is supposedly locked, and both the automation and the fast global control are up at the time. The fast LSC_ENABLE flag goes down at 06:10:10.03 UTC, so apparently not as a consequence of a cm command. ITF_LOCK enters the DOWN state at 06:10:11 UTC sharp from the DAQ/SMS point of view, but actually at 06:10:13.00 UTC according to the logs.
Looking at the zoomed plots of the event, we can see that the fast shutter behaviour was triggered by some glitch in either DARM or DARM/end mirrors correction signals, before the B1 saturation and the shutter closing.
From the automation point of view, the weird request is seen by ITF_LOCK and I suspect it happens more frequently than expected, but we see it if happens as the last one, so the one which sticks, while the first (the good one, in this case) is overwritten:
2025-07-03T06:10:13.008Z ITF_LOCK JUMP target: DOWN
2025-07-03T06:10:13.008Z ITF_LOCK [LOW_NOISE_3.exit]
2025-07-03T06:10:13.008Z ITF_LOCK STALLED
2025-07-03T06:10:13.537Z ITF_LOCK JUMP: LOW_NOISE_3->DOWN
2025-07-03T06:10:13.537Z ITF_LOCK calculating path: DOWN->LOW_NOISE_3
2025-07-03T06:10:13.544Z ITF_LOCK new target: FMODERR_TUNED
2025-07-03T06:10:13.554Z ITF_LOCK executing state: DOWN (1)
2025-07-03T06:10:13.555Z ITF_LOCK [DOWN.enter]
2025-07-03T06:10:13.575Z USERINPUT REQUEST: olserver52.virgo.infn.it virgorun old=LOW_NOISE_3 new=LOW_NOISE_3
2025-07-03T06:10:13.585Z USERINPUT REQUEST: olserver52.virgo.infn.it virgod old=LOW_NOISE_3 new=LOCKED_PRWI
2025-07-03T06:10:13.659Z ITF_LOCK in DOWN main
2025-07-03T06:10:13.674Z ITF_LOCK ['ITF_LOCK', 'DRMI_3F', 'CARM_SET_STEP1', 'CARM_SET_STEP2', 'CARM_SET_STEP3', 'CARM_NULL_3F', 'CARM_NULL_1F', 'DITHERING', 'SHUTTER', 'DC_READOUT', 'LOW_NOISE_1', 'LOW_NOISE_2', 'LOW_NOISE_3']
2025-07-03T06:10:14.008Z ITF_LOCK REQUEST: LOCKED_PRWI
It is also seen by CALI, and this is what should not happen, as in the DOWN state of CALI we clearly state to release ITF_LOCK from management (nodes.release(), and apparently it does, from the GUI point of view):
2025-07-03-06h10m13-UTC>INFO...-Unstalling ITF_LOCK
2025-07-03-06h10m13-UTC>WARNING-[DOWN.run] ezca: V1:GRD-ITF_LOCK_REQUEST => LOCKED_PRWI
In some cases, like yesterday, this happens with CALI complaining about connections errors:
2025-07-02-11h52m19-UTC>INFO...-Memory Used increase, cur. 2411364.00(KB), inc. 8.00(KB)
2025-07-02-12h56m41-UTC>INFO...-Unstalling ITF_LOCK
2025-07-02-12h56m41-UTC>WARNING-[DOWN.run] ezca: V1:GRD-ITF_LOCK_REQUEST => LOCKED_PRWI
2025-07-02-12h56m41-UTC>WARNING-[DOWN.run] USERMSG 0: EZCA CONNECTION ERROR: Any Other Error: Invalid REQUEST: LOCKED_PRWI
2025-07-02-12h59m41-UTC>INFO...-Memory Used increase, cur. 2411368.00(KB), inc. 4.00(KB)
2025-07-02-12h59m43-UTC>INFO...-Memory Used increase, cur. 2411520.00(KB), inc. 152.00(KB)
I will look more at the problem as soon as possible, one possibility is how the "unstalling" of nodes work, as it could be the actor that tries to re-instate a management that doesn't exist anymore, but this is still unclear. Another possibility is that we do the un-management wrong in some way (this is the only place where we un-manage nodes).
also the B4 12MHz show a correlation with the range. Moreover also the coupling of frequency noise has a strong correlation with the BNS, but in a strange way... the higher is the coupling the better is the range (figure 2 and 3). The origin of the trend has not be found yet
Last unlock at GPS time 1435519088 is due to Metatron glitch requesting LOCKED_PRWI state, same as this morning's only unlock at 06:10UTC.
This morning the ITF unlocked once at 6:10UTC with this request to ITF_LOCK:
2025-07-03T06:10:13.585Z USERINPUT REQUEST: olserver52.virgo.infn.it virgod old=LOW_NOISE_3 new=LOCKED_PRWI
???, it relocked at the first attempt and science mode started at 7:03UTC, science mode stopped one minute at 8:12UTC to change the NI etalon set pint.
ISC
- 8:12UTC: NI etalon set from 20.24 to 20.34, 54000 sec ramp time.
Yesterday we observed an other strange increase of the Range
the interesting thing is that is correlated with the inverse of the power on B1p, as for the event of the 20th of Jun. See figure 1 and 2. this time no correlation with a kick on SR tx.
to be understood.
also the B4 12MHz show a correlation with the range. Moreover also the coupling of frequency noise has a strong correlation with the BNS, but in a strange way... the higher is the coupling the better is the range (figure 2 and 3). The origin of the trend has not be found yet
I extended the analysis of the BNS range fluctuations over the past two days to include all _mean and _rms channels from the trend frames (~15k in total). The most correlated channels are those related to B5, i.e., the beam at the dark port before the Output Mode Cleaner. This is probably not informative.
Other notable channels include: V1:Sc_BS_CMRF, the control of the Common Mode Rejection Factor at the Beam Splitter, and V1:Sa_WI_F0_ACC_X, the X-axis acceleration at Filter 0 of the West Input.
Attached:
- The text file with the full ranking of channels showing a Pearson correlation greater than 0.2 (in absolute value),
-
and a few plots of the most correlated channels.
ITF found in LN3. SCIENCE mode. It kept the lock for the whole shift.
Guard tour (UTC):
20:59 - 21:35
23:10 - 23:40
01:07 - 01:43
03:11 - 03:57
