Today after the work at  UTA in DET  (# 64079) the squeezed unlocked due to the change of th ehumidity condition (FIG1). I waited for humidity condition recovery to check and try to relock the MZ. It happened at about 10 UtC. I started to check about at 20:13.

Nevertheless on the signal it was and it is present an oscillation with for me not understood origin (FIG2) un. I tried to check the origin but I was not able to find it and understand. I leave the system with the MZ unlocked and tomorrow morning I will contact Henning to check.

Note that the ITF unlock happend when I put in pause the SQZ MAIN automation to work on SQZ system. Maybe it is my fault (FIG3)

ITF found locked in LN3; the Squeezing was not working for temperature transient started in the morning due to the planned work at the UTA.

ITF was in Science but the GWISTAT page was displaying VIRGO in down; at the daily meeting it was discovered that it was due to a bug introduced by Hrec connected on FbmFE.

At 13:45 UTC Hrec process was restarted by CAL team with the previous version; this fixed the issue.

At 14:00 UTC ITF in Calibration to perform the daily calibration.

At 14:14 UTC ITF in Commissioning mode for New filters for marionette reallocation; activity concluded at 18:39 UTC.

At 18:39 UTC ITF in Science mode in LN3; because of the thermal transient in in progress in DET LAB the Squeezing could not be engaged.

At 20:30 UTC the ITF unlocked; relock in progress.

DET

At 16:10 UTC DET_Main went in unknown state after an unlock; the ITF was put in NI single bounce for safety and then the situation recovered with the procedure provided by Romain.

HVAC

The work at the HVAC DET was completed around 17:30 UTC; temperature and humidity inside the standard values.

QNR

Martina tried to recover the Squeezing but she was not able to do it; investigation is in progress.

Software

At 16:48 UTC BACnetServer restarted because it was not providing data.

Today, first of all, we made a test moving the lock from ITMs marionettes to ETMs marionettes. It worked as expected (a better balancing would be required), but it unlocked with the usual 1.8 Hz oscillation when the old filters for MIR/MAR splitting were engaged. Since this fact exclude that the oscillation depends on some strange coupling with DRMI dofs, we checked again other possibilities ad we found an oscillation of CARM SLOW (fig 1). We had excluded this possibility because the first time the problem was observed, CARM SLOW appeared stable. Probably we were wrong since the beginning and the right thing to do was to work on CARM SLOW loop. Anyway, it is not clear at all why a change of actuators - supposed to be transparent in terms of loop gain - should make CARM SLOW unstable. We decided to postpone an activity on that loop, because we need more time to understand better this issue and because we should also have a look on its noise projection.

For the moment, we improved the current reallocation strategy implementing a new High-Pass filter, which is a bit more efficient everywhere and shows no issue at 1.8 Hz (fig 2). We also validated the direct transition to LN2, gaining a factor of 2 in distance from saturation during the lock acquisition. This transition is currently used by default.

This morning, from 7h00 utc to 8h10 utc, we have added the channels slow_shutter_OMC1_T1_slow and slow_shutter_OMC1_T1_fast in the SDB1_OMC process. They will be used from now to monitor the state of the OMC slow shutter.

The channel slow_shutter_OMC1_T1_fast monitors the closing of the fast shutter (see Fig.1) which is presently done with the "MOVETOLIMIT" command at a speed of 1700 steps/s. The RMS of this channel is now used as observable in DetMoni to check if the shutter motor has stopped after the closing which is supposed to take place only when the ITF is DOWN in standard operations. Thus, if the ITF_LOCK index is above 1 and if the RMS of slow_shutter_OMC1_T1_fast indicates that the shutter is still running, a red flag will be triggered in the DMS.

The channel slow_shutter_OMC1_T1_slow monitors the opening of the fast shutter (see Fig.2) which is presently done with the "MOVEREL" command at a speed of  666 steps/s. The RMS of this channel is now used as observable in DetMoni to check if the shutter motor has stopped after the opening which is supposed to take place only at index 109 of ITF_LOCK. Thus, if the ITF_LOCK index is above 109 and if the RMS of slow_shutter_OMC1_T1_slow indicates that the shutter is still running, a red flag will be triggered in the DMS.

Since we had to start/stop the SDB1_OMC process several times for these operations, we verified at the end of the activity that the demodulation phase of the OMC locking signal was still correctly tuned (Fig.3).

The shift was dedicated to maintenance, here a list of the activity reported to the control room:

- standard Vacuum Refill from 6:00UTC to 10:00UTC (VAC Team);
- cleaning of central and mod cleaner buildings (Gargiulo with external firm: from 6:00UTC to 10:00UTC);
- HVAC: weekly checks and extraordinary maintenance fo the DET area UTA (Soldani 6:15UTC to 10:00UTC);
- SUSP: NE local control maintenance (Majorana from 7:55UTC to 9:40UTC);
- INJ: check of IMC working point and position from 6:22UTC to 6:32UTC;
- DET: OMC shutter driver replacement (#64070);
          OMC Lock in single bounce configuration from 6:49UTC to 6:55UTC;
          OMC scan in single bounce configuration from 7:03UTC to 7:24UTC
- TCS: chiller check and refill (Gargiulo 6:45UTC);
          thermal camera reference from 8:13UTC;
          power checks and power adjustment done by Ilaria;

all the activity completed at 10:00UTC, ITF relocked at the first attempt; the work on detection area air conditioning system created a big transient on the temperature (see plot) it prevented to use the squeezing that is suffering of the change of temperature (QNR on-call informed), for the time being we go on without the squeezing injected.

Air Conditioning
BACnetServer stopped and restarted at 9:57UTC because of missing data.

An offset has been added to each DAC output of the boards used to drive the MIR actuation of ITMs. Those offsets are required to compensate the small DC values provided by the driver when no correction is sent to the DACs. If those values are not compensated, the engagement of the series resistors during the transition to LN1 produces a series of glitches in DARM. The glitches are small enough and they don't unlock when the ETM drivers are in LN1, but are too large for LN2. In order to allow the direct transition to LN2 and help the lock acquisition in bad weather condition, the removal of the glitches was mandatory. As we can see in the attached figure, the lock acquisition done after the addition of the DC compensation was quite smooth, compared to a standard one. Now we are ready to test again the direct transition to LN2.

The following report has been submitted to the On-call interface.

On-call events -> Air Conditioning

Title: Water leakage warning.

Author(s): andreazzoli

Details:

I received four calls from the control room between 00:51 and 01:15.
I did not hear the calls.
At 08:02 I called the control room and spoke to the operator.
Soldani Davide had intervened during the night.
I decided to intervene for a short check to verify what had happened and if possible take action in order to correct the problem.

* Note that any files attached to this report are available in the On-call interface.

The following report has been submitted to the On-call interface.

On-call events -> Air Conditioning

Title: Water leakage warning.

Author(s): andreazzoli

Details:

I received four calls from the control room between 00:51 and 01:15.
I did not hear the calls.
At 08:02 I called the control room and spoke to the operator.
Soldani Davide had intervened during the night.
I decided to intervene for a short check to verify what had happened and if possible take action in order to correct the problem.

* Note that any files attached to this report are available in the On-call interface.

Today, around 8h55 UTC, I changed the config of Hrec to connect it to FbmFE instead of FbmAlp and use the science mode flag DQ_META_ITF_Mode_FE
instead of DQ_META_ITF_Mode and use the lock index flag META_ITF_LOCK_index_FE instead of META_ITF_LOCK_index.
Since 20/04 8h30 UTC, this new configuration was tested on HrecN test process and showed a decrease of the mean latency and an independence
with respect to the latency glitches present on FbmAlp (see plot1).
To do this connection of Hrec to FbmFE, I had to install a new version of Hrec code v5r13 in virgoApp.

I already made this logbook entry as 64034, but it disappeared. So I put it here again:

Over the long lock that started in the night of 18/04 to 19/04, we can see a correlated trend between the BNS range which starts dropping after 19/04 10h UTC
and the behaviour of the DCP frequency estimated by Hrec for the BS mirror and also the behaviour of the modulus of hrec/hinj at 77.5 Hz.
On the same plot, we can see that after 19/04 10h UTC, the optical gain estimated by Hrec is fluctuating more (as well as the BNS range)
and that the variations of the optical gain (estimated with the calibration lines around 70 Hz) are well correlated with the variations of hrec/hinj modulus at 77.5 Hz
over le ss than an hour time scale.
The trends over those 18 hours of lock may be linked to the etalon scan.
The larger fluctuations of BNS range, optical gain or hrec/hinj after 19/04 10h UTC have an other explanation.
Finding it would allow to improve the stability of the BNS range and of the bias hrec/hinj.

Today between 8:00 UTC and 9:00 UTC we have done two tests to investigate the issue of the OMC slow shutter with a broken stop.

Stop and start the driver. This did not resolve the issue, the shutter when opening still runs without stopping when arriving to the end. We have stopped it manually after 1 minute.

Replace the driver with a spare, serial number FT4ZNKQD. The configuration of the rotation process was updated with the new serial number. The issue remained the same. The translation stage of the shutter still doesn't stop when closing.

All these actions were performed with DET_MAIN in pause and the interferometer in single bounce.

In conclusion the driver is not the issue, and probably the in-vacuum translation stage has one of the two stops that is broken.

Upon arrival I found ITF just relocked and in Science Mode.
Unlocks:
- 23:19 UTC (?). Relocked at first attempt. 
- 01:32 UTC probably due to the seismic activity in Taiwan. Relockeed at second attempt at 03:36 UTC. 

Guard Tour (UTC)
21:06 - 21:34
23:08 - 23:34
01:04 - 01:30
03:15 - 03:45
 

https://gracedb.ligo.org/superevents/S240422ed/view/

Today the lock acquisition was very troublesome due an excess of noise on the West Green beam. We tried to check the coupling of the fibers on the DAQ room, but it didn't solve the problem. Then we went to the end building and tried to improve the fiber coupling by moving the fibers in the rack. We found a position where the noise was reduced. Still there is some green power missing, so tomorrow some more systematic alignment work will be done, in order to better align the green in the SHG box. After this action the lock acquisition worked properly.

Today, I found ITF unlocked with WI and WE guardian opened. I closed the guardian on both and realigned the mirrors to have back the flashes on B8, P. Ruggi used the dumper on WI Y to reduce the excitations of the mirror after the opening of the controls. I then aligned the CITF and restarted the locking acquisition. 15:30-15:40 UTC The WE guardian opened again. I properly closed and restored the position of the WE mirror that went off. M. Boldrini, J. Casanueva, D. Bersanetti, and M. Was are working on the ITF recovery. We discovered some problems, the first one was an issue with the WE ALS green (fixed by J. Casanueva and M. Boldrini with C. De Rossi from remote), second problem was with Ch e Das WI by mistake a flipper was left closed, I re-opened at 19:05UTC with the guidance of I. Nardecchia and the third problem was on PR_TX gain adjustments (D. Bersanetti and M. Boldrini). We had an 18:26 UTC earthquake in 6.3 Taiwan and an 18:32 UTC earthquake in 6.4 Taiwan. 20:50 UTC ITF back in SCIENCE. 

Yesterday's situation has been restored: PR_TX is reset to 2.2 in CARM_NULL_1F and is set to 2.2 as well in LOW_NOISE_1.

The gain change in LOW_NOISE_1 is kept in Metatron's code, but is currently doing nothing.

Around 19:00 LT, I discovered that the WI flip mirror used to acquire the thermal camera images was closed. This means that the CH and DAS were blocked on the bench since last night.

Gianmatteo re-opened it at 19.05 UTC.

A few of today's unlocks, after the wind settled down, have been caused by low gain oscillation for PR_TX (Fig.1).

Increasing the gain in CARM_NULL_1F caused an unlock soon after the transition, so we introduced a gain change in ACQUIRE_LOW_NOISE_1 (line 5232 and 5275 in ITF_LOCK.py).

The modifications can easily be reverted or made transparent (by simply setting this gain to the same value in CARM_NULL_1F and LOW_NOISE_1) if need be.

6:00 UTC: ITF found in BAD_WEATHER due to high wind and sea activity.

At 10:13 UTC I removed the mode BAD_WEATHER because the related flags WIND and SEA on the DMS were not red anymore but unfurtunately the lock acquisition continued to fail at various state; Maddalena checked from the control room.

After a check we discovered a change in the WI CH power that happend last night thus, at 10:30 UTC, I set TROUBLESHOOTING and we contacted Ilaria to restore the good value of the powers. This problem was not reported on the DMS thus a readjustment of the thresholds is needed.

After this change the interferometer continued to unlock also due to a further worsening of the weather conditions: strong storm with lightning and thunder.

The commissioning team present in control room is investigating at the problems.

ALS

As reported in the previous shift the DMS is reporting a problem with WE ALS Laser; this morning I informed the experts by mail. At 12:00 UTC I did not receive any answer thus we decided to contact the ALS on-call; unfurtunately he is not available to look at the problem.

SUSP

At around 12:50 UTC WE and WI LC were opened by the guardian after an unlock. I tried to close the loops but the new GUI interface did not work; for this reason I contacted the SUSP on-call. Investigation on the problem in progress from control room.

These events are likely correlated  to the burst of jobs that have been automatically submitted to the HTCondor farm by the DQR and detchar pipelines some time before, triggered by several H1 GstLAL alerts.

Between 4:20 UTC and 6:20 UTC about 1600 of these jobs have run causing a peak of more than 150MB/s nfs read traffic and consequent overload  of the fs01 fileserver where /virgoLog is hosted.

Since RTPCs still mount /virgo, /virgoApp and /cvmfs via NFS from fs01 instead of CVMFS, they may have been impacted by the corresponding slowdown which however has not provoked any visible errors at the Operating System level.

The altitude was also very low at the closest point (400ft --> 120mt).

Sometimes we can find data with BNS range stable above 56 Mpc for several minutes. This is the best level we can reach since several weeks. In march we had something better: for instance, on March 8th the range was 58.9 Mpc, averaged over 10 min.

In order to understand where those 2 Mpc have been lost, the two sensitivities are compared in fig 1. In order to understand better, the two cumulative BNS ranges have been computed and subtracted (fig 2).

It is clear enough that the mystery noise is not involved: a change of its level should be well visible in the region 80 - 120 Hz, where instead the sensitivity is equal. The bigger drop is below 80 Hz, both because an increase of the broadband noise and the presence of new structures. If we want to reach the psychological threshold of 60 Mpc, this region needs to be better studied.

The following report has been submitted to the On-call interface.

On-call events -> Air Conditioning

Title: water leak in COB officies conditioning pipe

Author(s): soldani

Details:

At the request of the In shift-Operator A.Magazzu', I reached the site to stop the water leak from the air conditioning pipe for the offices on the ground floor of the COB.
The relevant shutters were closed and the company in charge of maintenance was contacted, which this morning (22/04) is already working to replace the affected section of pipe (whose installation dates back to 1996) because its state of conservation does not allow a repair that gives a guarantee of reliability over time.
If there are no unforeseen events, the restoration work will be completed the same day.

* Note that any files attached to this report are available in the On-call interface.

The following report has been submitted to the On-call interface.

On-call events -> Injection System

Title: pmc pzt corrections high

Author(s): derossi

Details:

I was called by the operator because the flag PMC PZT was red.
We had set the flag red when the PMC PZT corrections become larger than 4,5V because in this conditions the lock would probably not last more than a few hours and it could be worth to manually unlock the PMC to reset the corrections (during the lock the corrections are likely to increase until 7V, causing the PMC unlock). Since is not something which prevents the ITF to lock we changed the color of the flag in the DMS: yellow from 4,5V to 6,5 and red starting from 6.5V.

* Note that any files attached to this report are available in the On-call interface.

Upon arrival I found ITF relocking after the earthquake.
Relocking in progress but ITF very noisy due to the high wind and sea activity increasing (plot3). The bad weather conditions prevent a stable lock for the whole shift.

ALS
I noticed that after the intervention at WE on ALS laser amplifier, the Green Power and the IR_PSTAB are not in a good working point (plot2).

SBE
After one of the unlock events SNEB Horizontal loop opened. Properly closed.

TCS
During a relock the CH TCS_CO2..REL3 got stuck at 1. I act ON Up via TTL_CO2.

Guard Tour (UTC)
21:06 - 21:34
22:55 - 23:25
01:04 - 01:50
03:07 - 03:39 

ISYS
(22-04-2024 04:00 - ) From remote
Status: On-going
Description: PSL_PMC_PZT out of range (plot1). Normally after an unlock it resets but not this night (?)
DMS comment: PMC_PZT_likely_to_drift_away.