The following report has been submitted to the On-call interface.

On-call events -> Air Conditioning

Title: Water leakage warning.

Author(s): andreazzoli

Details:

I received four calls from the control room between 00:51 and 01:15.
I did not hear the calls.
At 08:02 I called the control room and spoke to the operator.
Soldani Davide had intervened during the night.
I decided to intervene for a short check to verify what had happened and if possible take action in order to correct the problem.

* Note that any files attached to this report are available in the On-call interface.

Today, around 8h55 UTC, I changed the config of Hrec to connect it to FbmFE instead of FbmAlp and use the science mode flag DQ_META_ITF_Mode_FE
instead of DQ_META_ITF_Mode and use the lock index flag META_ITF_LOCK_index_FE instead of META_ITF_LOCK_index.
Since 20/04 8h30 UTC, this new configuration was tested on HrecN test process and showed a decrease of the mean latency and an independence
with respect to the latency glitches present on FbmAlp (see plot1).
To do this connection of Hrec to FbmFE, I had to install a new version of Hrec code v5r13 in virgoApp.

I already made this logbook entry as 64034, but it disappeared. So I put it here again:

Over the long lock that started in the night of 18/04 to 19/04, we can see a correlated trend between the BNS range which starts dropping after 19/04 10h UTC
and the behaviour of the DCP frequency estimated by Hrec for the BS mirror and also the behaviour of the modulus of hrec/hinj at 77.5 Hz.
On the same plot, we can see that after 19/04 10h UTC, the optical gain estimated by Hrec is fluctuating more (as well as the BNS range)
and that the variations of the optical gain (estimated with the calibration lines around 70 Hz) are well correlated with the variations of hrec/hinj modulus at 77.5 Hz
over le ss than an hour time scale.
The trends over those 18 hours of lock may be linked to the etalon scan.
The larger fluctuations of BNS range, optical gain or hrec/hinj after 19/04 10h UTC have an other explanation.
Finding it would allow to improve the stability of the BNS range and of the bias hrec/hinj.

Today between 8:00 UTC and 9:00 UTC we have done two tests to investigate the issue of the OMC slow shutter with a broken stop.

Stop and start the driver. This did not resolve the issue, the shutter when opening still runs without stopping when arriving to the end. We have stopped it manually after 1 minute.

Replace the driver with a spare, serial number FT4ZNKQD. The configuration of the rotation process was updated with the new serial number. The issue remained the same. The translation stage of the shutter still doesn't stop when closing.

All these actions were performed with DET_MAIN in pause and the interferometer in single bounce.

In conclusion the driver is not the issue, and probably the in-vacuum translation stage has one of the two stops that is broken.

Upon arrival I found ITF just relocked and in Science Mode.
Unlocks:
- 23:19 UTC (?). Relocked at first attempt. 
- 01:32 UTC probably due to the seismic activity in Taiwan. Relockeed at second attempt at 03:36 UTC. 

Guard Tour (UTC)
21:06 - 21:34
23:08 - 23:34
01:04 - 01:30
03:15 - 03:45
 

https://gracedb.ligo.org/superevents/S240422ed/view/

Today the lock acquisition was very troublesome due an excess of noise on the West Green beam. We tried to check the coupling of the fibers on the DAQ room, but it didn't solve the problem. Then we went to the end building and tried to improve the fiber coupling by moving the fibers in the rack. We found a position where the noise was reduced. Still there is some green power missing, so tomorrow some more systematic alignment work will be done, in order to better align the green in the SHG box. After this action the lock acquisition worked properly.

Today, I found ITF unlocked with WI and WE guardian opened. I closed the guardian on both and realigned the mirrors to have back the flashes on B8, P. Ruggi used the dumper on WI Y to reduce the excitations of the mirror after the opening of the controls. I then aligned the CITF and restarted the locking acquisition. 15:30-15:40 UTC The WE guardian opened again. I properly closed and restored the position of the WE mirror that went off. M. Boldrini, J. Casanueva, D. Bersanetti, and M. Was are working on the ITF recovery. We discovered some problems, the first one was an issue with the WE ALS green (fixed by J. Casanueva and M. Boldrini with C. De Rossi from remote), second problem was with Ch e Das WI by mistake a flipper was left closed, I re-opened at 19:05UTC with the guidance of I. Nardecchia and the third problem was on PR_TX gain adjustments (D. Bersanetti and M. Boldrini). We had an 18:26 UTC earthquake in 6.3 Taiwan and an 18:32 UTC earthquake in 6.4 Taiwan. 20:50 UTC ITF back in SCIENCE. 

Yesterday's situation has been restored: PR_TX is reset to 2.2 in CARM_NULL_1F and is set to 2.2 as well in LOW_NOISE_1.

The gain change in LOW_NOISE_1 is kept in Metatron's code, but is currently doing nothing.

Around 19:00 LT, I discovered that the WI flip mirror used to acquire the thermal camera images was closed. This means that the CH and DAS were blocked on the bench since last night.

Gianmatteo re-opened it at 19.05 UTC.

A few of today's unlocks, after the wind settled down, have been caused by low gain oscillation for PR_TX (Fig.1).

Increasing the gain in CARM_NULL_1F caused an unlock soon after the transition, so we introduced a gain change in ACQUIRE_LOW_NOISE_1 (line 5232 and 5275 in ITF_LOCK.py).

The modifications can easily be reverted or made transparent (by simply setting this gain to the same value in CARM_NULL_1F and LOW_NOISE_1) if need be.

6:00 UTC: ITF found in BAD_WEATHER due to high wind and sea activity.

At 10:13 UTC I removed the mode BAD_WEATHER because the related flags WIND and SEA on the DMS were not red anymore but unfurtunately the lock acquisition continued to fail at various state; Maddalena checked from the control room.

After a check we discovered a change in the WI CH power that happend last night thus, at 10:30 UTC, I set TROUBLESHOOTING and we contacted Ilaria to restore the good value of the powers. This problem was not reported on the DMS thus a readjustment of the thresholds is needed.

After this change the interferometer continued to unlock also due to a further worsening of the weather conditions: strong storm with lightning and thunder.

The commissioning team present in control room is investigating at the problems.

ALS

As reported in the previous shift the DMS is reporting a problem with WE ALS Laser; this morning I informed the experts by mail. At 12:00 UTC I did not receive any answer thus we decided to contact the ALS on-call; unfurtunately he is not available to look at the problem.

SUSP

At around 12:50 UTC WE and WI LC were opened by the guardian after an unlock. I tried to close the loops but the new GUI interface did not work; for this reason I contacted the SUSP on-call. Investigation on the problem in progress from control room.

These events are likely correlated  to the burst of jobs that have been automatically submitted to the HTCondor farm by the DQR and detchar pipelines some time before, triggered by several H1 GstLAL alerts.

Between 4:20 UTC and 6:20 UTC about 1600 of these jobs have run causing a peak of more than 150MB/s nfs read traffic and consequent overload  of the fs01 fileserver where /virgoLog is hosted.

Since RTPCs still mount /virgo, /virgoApp and /cvmfs via NFS from fs01 instead of CVMFS, they may have been impacted by the corresponding slowdown which however has not provoked any visible errors at the Operating System level.

The altitude was also very low at the closest point (400ft --> 120mt).

Sometimes we can find data with BNS range stable above 56 Mpc for several minutes. This is the best level we can reach since several weeks. In march we had something better: for instance, on March 8th the range was 58.9 Mpc, averaged over 10 min.

In order to understand where those 2 Mpc have been lost, the two sensitivities are compared in fig 1. In order to understand better, the two cumulative BNS ranges have been computed and subtracted (fig 2).

It is clear enough that the mystery noise is not involved: a change of its level should be well visible in the region 80 - 120 Hz, where instead the sensitivity is equal. The bigger drop is below 80 Hz, both because an increase of the broadband noise and the presence of new structures. If we want to reach the psychological threshold of 60 Mpc, this region needs to be better studied.

The following report has been submitted to the On-call interface.

On-call events -> Air Conditioning

Title: water leak in COB officies conditioning pipe

Author(s): soldani

Details:

At the request of the In shift-Operator A.Magazzu', I reached the site to stop the water leak from the air conditioning pipe for the offices on the ground floor of the COB.
The relevant shutters were closed and the company in charge of maintenance was contacted, which this morning (22/04) is already working to replace the affected section of pipe (whose installation dates back to 1996) because its state of conservation does not allow a repair that gives a guarantee of reliability over time.
If there are no unforeseen events, the restoration work will be completed the same day.

* Note that any files attached to this report are available in the On-call interface.

The following report has been submitted to the On-call interface.

On-call events -> Injection System

Title: pmc pzt corrections high

Author(s): derossi

Details:

I was called by the operator because the flag PMC PZT was red.
We had set the flag red when the PMC PZT corrections become larger than 4,5V because in this conditions the lock would probably not last more than a few hours and it could be worth to manually unlock the PMC to reset the corrections (during the lock the corrections are likely to increase until 7V, causing the PMC unlock). Since is not something which prevents the ITF to lock we changed the color of the flag in the DMS: yellow from 4,5V to 6,5 and red starting from 6.5V.

* Note that any files attached to this report are available in the On-call interface.

Upon arrival I found ITF relocking after the earthquake.
Relocking in progress but ITF very noisy due to the high wind and sea activity increasing (plot3). The bad weather conditions prevent a stable lock for the whole shift.

ALS
I noticed that after the intervention at WE on ALS laser amplifier, the Green Power and the IR_PSTAB are not in a good working point (plot2).

SBE
After one of the unlock events SNEB Horizontal loop opened. Properly closed.

TCS
During a relock the CH TCS_CO2..REL3 got stuck at 1. I act ON Up via TTL_CO2.

Guard Tour (UTC)
21:06 - 21:34
22:55 - 23:25
01:04 - 01:50
03:07 - 03:39 

ISYS
(22-04-2024 04:00 - ) From remote
Status: On-going
Description: PSL_PMC_PZT out of range (plot1). Normally after an unlock it resets but not this night (?)
DMS comment: PMC_PZT_likely_to_drift_away.

Today, I found ITF in SCIENCE. 18.30 - 18:42 UTC I started the scheduled DF_DAILY calibration. 18:43 UTC ITF back in SCIENCE. 20:11 UTC An earthquake (5.5 Mexico) caused ITF to unlock. I realigned the CITF and restarted the locking acquisition. 20:59 UTC ITF unlocked at LOW_NOISE_2.

Other
VPM
18:43 UTC restarted CALI after a crash

" So probably the helicopter respected the NOTAM. "

I guess not

Nice idea. However the original restrictions - still enforced, as reported in the attached text of the forbidden areas of the latest release of the AIP publication (https://www.enav.it/servizi-online) - are cylinders of 2000 ft = 610 m of height but radius of 0.3 nautical miles = 1823 ft = 556 m. So probably the helicopter respected the NOTAM. 

Tonight a strong and clean PC_1 magnetic pulsation was received in both the underground magnetometers of Sos Enattos and the external magnetometer of Virgo.

Here its "sound" when accelerated and shifted in frequency 100 times.

 (credits: Renato Romero www.vlf.it)

This is a first (preliminary) look at the effect of acoustic injection into Hrec, performed on Thursday 18.

For each injection (at the 0.03, 0.04 and 0.05 V levels) the BNS range decreased.

Here the entire Hrec range from 10 to 1000 Hz.

There are regions growing at low frequency (around 120 and 170 Hz) and high frequency (the most relevant from 500 to 540 Hz).

Below 100 Hz nothing seems to have changed significantly.

ITF in Science mode for all the shift.

Guard tours (time in UTC)

From 6:00 to 6:30; from 8:00 to 8:30; from 10:00 to 10:30; from 12:00 to 12:30.

Upon reaching the site, I found the ITF locked in LOW_NOISE_3_SQZ and in Science Mode. At 22:12 UTC the Squeezing disengaged, bringing the ITF back in LOW_NOISE_3. ITF back in LOW_NOISE_3_SQZ from 22:14 UTC.
ITF left locked.

Guard Tour (UTC):
21:30 - 22:00
23:30 - 00:00
1:38 - 2:07
3:07 - 3:18

DAQ
The process ELE_QGBT was not working correctly, it was reported as not working on DMS and in Orange state on VPM. At 4:13 UTC I restarted the process to solve the issue.