Draft report during afternoon shift
Upon arrival I found ITF stable locked at LN3_SQZ (Science Mode).
Planned Calibration CALIBRATED_DF_PCAL planned for 18:30 UTC.
Draft report during afternoon shift
Upon arrival I found ITF stable locked at LN3_SQZ (Science Mode).
Planned Calibration CALIBRATED_DF_PCAL planned for 18:30 UTC.
Today, I found ITF in SCIENCE. 06:06 UTC ITF unlocked (probably caused by missing data, see attachment). I called the ALS on call A. van de Walle, to restart the WE ALS laser, He will contact C. De Rossi to come on-site to do it. 09:30 UTC C. De Rossi replaced the head of the WE green laser and SOA driver. I religned the CITF at 9:45 UTC and 10.32 UTC ITF back in SCIENCE.
No DMS events
Sub-system reportsOn-line Computing & Storage
Missing data 06:06:27-06:06:38 UTC and 06:16:29-06:16:41 UTC
Other
(20-04-2024 07:45 - 20-04-2024 10:10) On site
Status: Ended
Description: WE ASL laser off
Actions undertaken: C. De Rossi replaced SOA Driver and the head of the WE ALS laser
I have refined the analysis in the main entry to include more glitches and all the auxiliary channels measuring positions, velocities and accelerations, as described in this git issue. The results are contained in the attached text files. These confirm and expand what is already presented in the main entry, that is, a strong correlation of these glitches with the movement of the West End bench and suspension. Correlations with velocity and acceleration sensors are instead much smaller. They tend to point to the motion of parts of the North End building, but plotting the predictor frequencies superimposed to the spectrogram of Hrec shows in fact no convincing correlations. Example in figure 1.
The following report has been submitted to the On-call interface.
On-call events -> Interferometer Sensing & Control
Title: ALS WEB- SOA replacement
Author(s): derossi
Called at: 08:50, 20-04-2024, by: Sposito |
Remote intervention: Started: ; Ended: |
On-site intervention: Started: 09:45, 20-04-2024; Ended: 12:10, 20-04-2024 |
Status: Resolved |
Operator when issue resolved: Sposito |
Details:
No more green was produced at the WEB and after some investigations it was found out that it was a failure of the optical amplifier (SOA).
This morning the ITF unlocked and I was call to go there and replace it.
* Note that any files attached to this report are available in the On-call interface.
Around 2024-04-20-06h06m28 most of the TolmFrameBuidler servers were unable to transmit theirs frames : it s probably the cause of the ITF unlock
Below the message reported by some TolmFRameBuilder servers
The IT departement has been informed to check for a possible network switch issue
Upon reaching the site, I found the ITF locked in LOW_NOISE_3 and in Science Mode. It kept the lock for the whole shift.
Guard Tour (UTC):
21:06 - 21:36
22:55 - 23:25
1:33 - 2:02
3:50 - 4:20
DAQ
The process ELE_QGBT was not working correctly, it was reported as not working on DMS and in Orange state on VPM. At 1:05 UTC I restarted the process to solve the issue.
Today at about 14:30 UTC we accessed the NE building to inspect and lubricate the regulator of the pressurized air driving the cryogenic valve #01, that showed signs of malfunctioning. The intervention was concluded within about 10 min (see also operator's entry). This device will be replaced with a new one next Tuesday .
Upon arrival I found ITF stable in Science Mode (LN3_SQZ).
At 13:29 and at 13:33 UTC the SQZ unlocked, relocked at first attempt in both occasions (Science Mode set) but at third failure I contacted SQZ OnCall who checked on it. (Science Mode with ITF in LN3). The issue was related to the humidity level in Det Lab.
DET
13:39 UTC - DetMoni VPM process restarted under request of Gouaty.
At 14:24 UTC I set ADJUSTING in order to allow:
- ACS experts to fix the problem on DET Humidity. Activity concluded at 14:40 UTC. The steam producer was blocked. Soldani pointed out to me that the ENV DMS flag (ENV..SQZ_BOARD_HU) values are not "aligned" with the INF DMS flag (HVAC..DET_HU_IN).
- VAC experts to go at NE to check a CryoTrap valve. Activity concluded at 14:45 UTC.
- ALS Gosselin to reboot green ALS Laser but unfortunately the optic amplifier is not responding anymore. A replacement is planned for tomorrow morning -> AT NEXT UNLOCK WE ARE NOT ABLE TO RELOCK !
15:21 UTC - ITF again in Science Mode
CAL
In order to avoid the risk of ITF unlock, Was asked me to perform "only" 10 times the CALIBRATED_DF_DAILY.
19:00 UTC - ITF set in LN3 (CALIBRATION MODE) Also L1 and H1 DOWN.
- Calibrations started at 19:01 and concluded at 20:54 UTC
LN3_SQZ / Science Mode at 20:55 UTC.
Oncall events
SQZ
(19-04-2024 14:00 - 19-04-2024 14:25) From remote
Status: concluded
Description: The result of check of the SQZ failure was due to the humidity in DET area ENV..SQZ_BOARD_HU.
ACS
(19-04-2024 14:25 - 19-04-2024 14:45) On site
Status: concluded
Description: Det Humidity intervention by ACS Expert OnCall, steam producer restarted. Humidity values again in threshold.
ALS
(19-04-2024 13:00 - 19-04-2024 15:20) From remote
Status: concluded
Description: The issue was at level of optic amplifier. Aymeric contacted ISYS OnCall expert (C. DeRossi) who will come on site tomorrow morning to replace the device at WE with a spare one.
TCS
(19-04-2024 16:36 - 19-04-2024 17:00) From remote
Status: concluded
TCS_WI_CO2_PWROUT out of range. Nardecchia asked me to change the DMS thresholds.
Guard Tour (UTC)
17:01 - 17:40
19:01 - 19:28
ERRATA CORRIGE: the sidebands at 44 and 56 Hz are non-physical. These are just an aliasing artefact, consequence of an improper chunk-by-chunk downsampling to produce the resulting spectrogram.
Figure 1: fixed version of the spectrogram in the range 42-58 Hz.
Figure 2: animated GIF that compares the correct spectrogram and the wrong one in the main entry.
I've checked with some Q-scans if the transit of the helicopter caused glitches in Hrec.
Figure 1: During the first transit at 14:46 UTC no excess noise was visible in Hrec, while in the CEB microphone is clearly visible the sweep caused by the helicopter.
Figure 2: during the transit at 14:59 UTC some barely coincident noise is visible in Hrec and the CEB microphone. The frequency is not exactly the same, so this may have just happened by chance.
Figure 3: some excess noise but no evident correlation at the time of the transit at 15:11 UTC. Nothing evident during all the other passages of the helicopter.
We tested the sensitivity of the top-stage BS accelerometers at three different levels of acoustic injection (nominally 0.03 - 0.04 - 0.05 V).
Noise was filtered from 8 to 2000 Hz, but the actual noise produced was only from 20 Hz and up, as the speaker is unable to go lower.
The fourth injection (18:04:40 UTC - (80 -100) Hz, 300 sec, 0.04 V) aimed to produce the same level around the frequency of ~93 Hz (without generating out-of-band noise) where one of the horizontal BS F0 accelerometers (Sa_BS_F0_ACC_H3_500Hz) is usually excited during acoustic injections, in order to check whether the coupling is linear or not.
Unfortunately, the filter was too narrow, and the noise spectral density at 93 Hz was lower than expected.
However, we can say that generating a higher level by a factor of 2 (with no noise at lower frequencies) resulted in an increase of the peak by about a factor of 2 (difficult to estimate accurately). It appears that there is no upconversion and the system is "linear."
At first glance one might think of general coupling due to the external accelerometer cables running not too far from the magnetometers (as indeed they are), but there are high-level signals in the acceleration spectrum that have no coherence. If it were a coupling from the cable, it should be broadband.
We have updated the Shutter flag in the DMS (configuration of process DetMoni). In case the slow shutter motor remains stuck after step 109 of the lock acquisition, a red flag will be displayed.
We plan to further update the flag logic at the next maintenance to also perform some checks during the lock acquisition after the shutter closure.
Indeed around the time of the unlock (yesterday 15:22 UTC) there has been also an earthquake (M 5.6 - 9 km W of Sulusaray, Turkey, 15:11 UTC)
The unlock was likely due to it, as reported by A.Magazzu in https://logbook.virgo-gw.eu/virgo/?r=64020.
This makes difficult to disentangle if there has been aslo an impact from the helicopter.
ITF in SCIENCE for the whole shift.
Pending actionsOther
( - ) ALS WE Laser has to be switched back on at the first unlock
Yesterday an helicopter was seen flying close to the Virgo buildings. This was in the late morning but the ITF was not locked. And again in the afternoo (14:30 to 15:30) when the ITF was locked.
Some indication of the passages are in the first plot, showing the RMS of some microphones in the band around 50Hz.
Helicopther noise is seen also at NEB and WEB. The sound signal is visible the band up to 200Hz, louder around 30 Hz (this can bee seen in Figure 2).
To a first look no evident glitch is seen in Hrec.
Yet, the unlock (around 15:20) is associated to scattered light arches in hrec and in B8 photodiode. At that time, judgeing from the sound level in microphones, the helicopter was close to WEB.
A correlation of the two events is possible (Figure 3).
But I would advise a better look with more accurate tools.
Indeed around the time of the unlock (yesterday 15:22 UTC) there has been also an earthquake (M 5.6 - 9 km W of Sulusaray, Turkey, 15:11 UTC)
The unlock was likely due to it, as reported by A.Magazzu in https://logbook.virgo-gw.eu/virgo/?r=64020.
This makes difficult to disentangle if there has been aslo an impact from the helicopter.
I've checked with some Q-scans if the transit of the helicopter caused glitches in Hrec.
Figure 1: During the first transit at 14:46 UTC no excess noise was visible in Hrec, while in the CEB microphone is clearly visible the sweep caused by the helicopter.
Figure 2: during the transit at 14:59 UTC some barely coincident noise is visible in Hrec and the CEB microphone. The frequency is not exactly the same, so this may have just happened by chance.
Figure 3: some excess noise but no evident correlation at the time of the transit at 15:11 UTC. Nothing evident during all the other passages of the helicopter.
This is very interesting. My interpretation of to these results is:
Upon reaching the site I found the ITF relocking. After a brief alignment of the PR/SR Mirrors, LOW_NOISE_3_SQZ was reached at the first attempt at 13:50 UTC, ITF back in Science Mode from 15:32 UTC.
At 15:07 UTC I set the ITF in Commissioning Mode to allow the planned ISC activity, DIFFp set point error signal test, carried out by Mantovani. The work could not be concluded due to an unlock at 15:22 UTC due to an earthquake (M 5.6 - 9 km W of Sulusaray, Turkey, 15:11 UTC). After various attempts we were able to reach LOW_NOISE_3_SQZ at 16:53 UTC, and the second planned activity, ENV Acustic Injection, started (see report #64024).
ENV Activity concluded at 18:11 UTC and ITF back in Science Mode. A quick adjusting period, from 18:16 UTC to 18:17 UTC, was done to allow the switch off the Speaker PDU.
The ITF unlocked at 19:34 UTC, relock in progress.
Guard tour (UTC):
17:06 - 17:44
19:03 - 19:30
Prompted by Michal, I verified if the occurrence of Scattered Light glitches from External Injection Bench motion associated with bad weather conditions has changed after the recent intervention and improvement of the EIB ground noise correction: #63763.
I examined the conditions during the last three main events of bad weather and Scattered Light glitches: February 25-26, March 26-27, and April 1-2.
Figure 1: time series of the V1:ENV_CEB_SEIS_W_50Hz_rms_0.1_1Hz during the three examined periods. In all three cases, the level of ground motion in the CEB is comparable.
Figures 2, 3 and 4: glitchgrams for Hrec during the three examined periods. The bottom time series represent the glitch rates for those triggers with frequency at peak between 20 and 2048 Hz, and SNR > 6.5. One interesting thing to notice is the presence of additional, very loud glitches with frequency at peak of about 90-100 Hz, associated with the last period of bad weather. Investigations are ongoing to understand their origin and if they are related or not to the ground motion.
Figures 5, 6 and 7: same as above but for the V1:LSC_SRCL channel, which in previous entries (#63761 and , #63420) was observed with Scattered Light glitches most correlated with the motion of the EIB, represented by the channel V1:SBE_EIB_GEO_H2_200Hz. In the period April 1-2, almost no glitches are present above 20 Hz, while in the other two periods, the rate is very high and correlated with the ground motion. The fact that a similar reduction in the rate of glitches is not visible in Hrec suggests a different origin for them.
Figure 8: a few Scattered Light glitches present in LSC_SRCL and their prediction with the SBE_EIB_GEO_H2_200Hz channel, as described in #63761.
Figure 9: the Scattered Light visible in Hrec is mostly explainable by the motion of the West End mirror. Using the technique described in #63761, the most correlated channel found is Sa_WE_F0_X_500Hz. In the attached text file, the list of all channels with correlation larger than 20%. The reconstruction is not perfect due to the sparseness and irregular shape of the glitches.
This is very interesting. My interpretation of to these results is:
I have refined the analysis in the main entry to include more glitches and all the auxiliary channels measuring positions, velocities and accelerations, as described in this git issue. The results are contained in the attached text files. These confirm and expand what is already presented in the main entry, that is, a strong correlation of these glitches with the movement of the West End bench and suspension. Correlations with velocity and acceleration sensors are instead much smaller. They tend to point to the motion of parts of the North End building, but plotting the predictor frequencies superimposed to the spectrogram of Hrec shows in fact no convincing correlations. Example in figure 1.
We performed a set of four acoustic injections in CEB with the AcousticInjectionCEB VPM process:
1) 17:07:55 UTC - (8 - 2000) Hz, 600 sec, 0.03 V
2) 17:21:05 UTC - (8 - 2000) Hz, 600 sec, 0.04 V (25 min glitch at ~17:29:24 UTC)
3) 17:33:48 UTC - (8 - 2000 )Hz, 600 sec, 0.05 V
4) 18:04:40 UTC - (80 -100) Hz, 300 sec, 0.04 V
The directory path of output files is /virgoData/NoiseInjections/AcousticInjectionsO4/output. The list of file injections is reported below:
1) AcousticColored_NOISE_CEB_DetTerrace-1397495261.txt
2) AcousticColored_NOISE_CEB_DetTerrace-1397496052.txt
3) AcousticColored_NOISE_CEB_DetTerrace-1397496814.txt
4) AcousticColored_NOISE_CEB_DetTerrace-1397498666.txt
As Piernicola confirmed, the sidebands at 46.7 Hz and 53.3 Hz are the beating between 50 Hz and DIFFp_TX line at 3.3 Hz (purple curve).
Indeed, when the DIFFp_TX line was set to 4.8 Hz, there were two sidebands at 45.2 Hz and 54.8 Hz, respectively (blue curve).
This afternoon we performed a few tests on the OMC shutter with the ITF in NI single bounce in order to find a solution to the problem of OMC shutter getting stuck at the opening.
Fig.1 shows a test when sent a MOVETOLIMIT command via VPM to open the shutter. In this case the shutter motor get stuck, and we stopped it by sending the STOP MOTION command.
We have tested the opening of the shutter by sending some MOVEREL commands with a defined number of steps. We have found that with about 36000 steps the power on B1s reaches a plateau. Adding a bit of margin, we decided to perform 42000 steps to open the shutter. This is tested on Fig.2. We can see that it takes about 65 seconds to accomplish this motion.
We have then replaced the MOVETOLIMIT command by the MOVEREL command for the opening of the shutter in the class SHUTTER_OPENING of DET_MAIN. In the DET_MAIN.ini file, we have added the parameter nominal_open_steps = 42000, and we have updated the opening_duration to 80 seconds.
The opening and then closing of the shutter are tested with the automation on Fig.3. The closing is still performed with the MOVETOLIMIT command in order to be sure that we always start opening the shutter from the same position.
It was then possible to relock the ITF in Low Noise 3 at first trial.
The unlock due to bad weather was not associated to the limited range of the actuators, as happened other times. When the wind increased and reached 50 km/h in a few minutes, WE_MIR_CORR increased, but not enough to stay so close to the saturation (fig 1). The unlock happened during a clear drop of the sidebands, at the end of which some longitudinal loop was not very stable anymore (fig 2). The drop was associated to a misalignment of BS TY (fig 3) and PR_TY (fig 4). It is not clear to me which was the first which began to move, and what was the cause of this motion.
Upon arrival I found ITF in Science Mode.
ITF stable locked at LN3_SQZ (Science Mode) till 11:31 UTC when it unlocked due to the bad weather conditions. High wind activity.
According with Gouaty I set ITF in SINGLE_BOUNCE_NI in order to allow Romain on his investigation on OMC shutter and B1_PD3 noise.
TROUBLESHOOTING MODE set from 11:33 to 12:56 UTC. Activity concluded. Relock in progress...
NO DMS events Monitor to report
The unlock due to bad weather was not associated to the limited range of the actuators, as happened other times. When the wind increased and reached 50 km/h in a few minutes, WE_MIR_CORR increased, but not enough to stay so close to the saturation (fig 1). The unlock happened during a clear drop of the sidebands, at the end of which some longitudinal loop was not very stable anymore (fig 2). The drop was associated to a misalignment of BS TY (fig 3) and PR_TY (fig 4). It is not clear to me which was the first which began to move, and what was the cause of this motion.
Looking at the spectrogram of Hrec, we observed a peak at 47.5 Hz which arose on April 15th, ~01:00 UTC (between two locks), see Figure 1 and 2.
It could be related to the F7 crossbar mode of the NI tower that we measured by magnetic injections to be at 47.65 Hz (VIR-0300A-24).
In this case, it seems that the line has shifted slightly in frequency.