The following report has been submitted to the On-call interface.

On-call events -> Electricity

Title: ENEL GE1 Network Alarm

Author(s): dandrea

Details:

I received a call from the alarm system on the power grid about a lack of ENEL grid on the GE1 that powers the central building and the control building.
I intervened on the site and found the rescue generator set in operation, as it did not have the ENEL reference voltage.
A fuse had blown on the control board which I promptly replaced and the system returned to normal operation.

* Note that any files attached to this report are available in the On-call interface.

ITF found DOWN and in UPGRADING Mode.
All times are UTC.
Below the list of actiities comunicated to the Control Room.
07:08 - 07:35 PAY: NI electrical checks with payload integrated with the superattenuator, works on the platform (Gherardini, Pinto, #69369).
07:25 - 10:25 PAY: WI payload assembly in CEB Clean Room (Dicorato, Puppo, Ricci, Travasso).
13:37 - 13:57 Inspection of CEB IR Cameras (Boldrini, Spinicelli, #69375).

We surveyed the NI and WI tower to take note of possible safety issues concerning the viewports used for the PAY cameras.

We restored the position of the NI_Cam and enveloped both the camera and the viewport with a black aluminum sheet. That viewport has been secured and no light from the YAG will be able to exit the tower through it.

For the WI_Cam, the situation is a lot different (pictures 1,2). The camera is fixed on a metal bar, propped against the tower and resting directly on the SLED box. The viewport itself is completely exposed, which will be a safety concern once the YAG circulates again in the WI tower.

We'll take steps to address these concerns as soon as possible.

A comb of line spaced by 50Hz is present in the PSTAB spectrum, and the projection of those lines in the sensitivity show that the 250Hz line, and maybe some other are dominating the contribution of 50Hz harmonics to the sensitivity. The real solution will be to have fully differential calbing from the PSTAB photodiodes, instead of single ended calbing inside the vacuum as is currently the case. In the mean time, a feed forward of the 50Hz and its harmonics could be tried to reduce those lines. 

https://git.ligo.org/virgo/commissioning/commissioning-tasks/-/work_items/8

Paola has kindly explained how the 50Hz feed-forward into DARM is implemented in the DSP and I have replicated the method in Acl. The environmental channel is bandpassed around 50Hz and then a variable delay+gain filter is applied to the environmental channel with two continuously updated parameters. The filter gain is a simple product by a parameter, and the dephasing is formed by implementing an (s-s0)/(s+s0) filter manually in Acl retaining in a separate channel the previous value of a channel.

To obtain error signals for those two parameters the PSTAB channel after after adding the  feed-forward correction  is band passed at 50Hz, and multiplied by the environmental channel band passed around 50Hz and filtered by the dephasing filter. After low passing the result this yields the error signal for the gain parameter of the feedforward. 

To obtain the delay parameter error signal, the environmental channel is first filtered by a pole at 1Hz, to dephase it by 90 degrees before multplying it with the PSTAB channel band passed at 50Hz and low-passing. In this way the other quadrature of the demodulation of the PSTAB by the environmental channel is obtained.

To test the Acl implementation Alain has setup a test process Acl17_INJ in the test instance of VPM: https://vpm.virgo.infn.it:40010/main.html?subsystem=Tests&process=Acl17_INJ

To access the data from it one needs to change the Cmdomain using the command "cmdomain CascinaTest" before starting dataDisplay, and then choosing the FbmMainTestUsers online server.

Figure 1 shows that after fixing many mistakes the 50Hz feed-back works in a test, reducing the 50Hz line in the target PSTAB channel by two orders of magnitude. FF_product_I/Q are the two error signals, while FF_gain and FF_z0 are the two variable parameters of the feed-forward filter. The adjustment of those filters is done using a simple pure integrator of the error signals, and the adjustment loop UGF is around 0.2Hz, I haven't tried to increase the UGF further.

One important aspect was to do the demodulation for calculating the FF_product_I/Q error signals after applying the dephasing signal to the environmental channel, so that the loop automatically tunes the demodulation phase. Initially the loop was not working at all, as the 50Hz line in the PSTAB channel was ~90 degree out of phase with the line in the environmental channel, which was swapping the two error signals.

This could be tested at some point on the actual PSTAB, starting from next week, to see if that idea works in practice. The code naming need to be first cleaned up, added into the ISYSnoise process to include it into the existing PSTAB_NOISE channel, and eventually replicated for all of the lines in the 50Hz harmanics comb.

During the recovery of the Injection system in May, we had to search manually for a new alignment of the IMC cavity, in order to continue with the all the other checks.

However, with the chosen position we had quite large offset on both DC QPD (MC transmission) and RF QPD (Dihedron reflection) in addition to the one on the longitudinal lock. Moreover, we usually apply offset on the erorr signals after the sensing matrix (IB_{TX, TY, TZ}, MC_{TX, TY}), thus generally mixing the offset on DC and RF quadrants.

As first step to improve the working point, we tried to zeroing the RF signals (INJ_IMC_QD_{NF,FF}_I_{H,V}) by manually mixing all the AA offset ( IB_{tx,ty,tz}_set and MC_{tx,ty}_set). In addition, we were also brought back the beam on the MC close to the position of last March (looking at INJ_MCT_{h,v}_err). The results are shown in fig. 1: reducing the offset on the RF quadrants reduced the frequency to amplitude coupling (new IMC_SET offset is now -1.5) and clearly improved the alignment of the RFC. In addition, not completely unexpected, there was a change on the PSTAB photodiodes, that induced a small improvement on the PSTAB correction noise.

We then modified the AA loop on ACL in order to better take into account the offset on the DC quadrant ({NF,FF}_AC{h,v}_SET, MCT_{h,v}_SET) and recomputed the AA matrix accordingly. We closed the AA in full bandwith at the end of the work (fig.2).

in the afternoon I worked again in the NI. I left at 19.30 LT

Between 11:00 and 13:00 utc, I tuned the offset of NCal temperature probes to have all of them at 22.5°C+-0.1°C when all rotors are off.

Then around 13:50, I switched all rotors ON to their default lines and closed the loops. 

Details:

Strictly speaking we shoud tune the parameters inside  /virgoData/VirgoOnline/SNEB_Fb.cfg and SWEB_Fb.cfg which set the parameters of the volts to temperature function.

But in good approximation, it is OK to modify the offset of the ADC to Volts conversion in /virgoData/VirgoOnline/SWEB_dbox_rack.cfg and /virgoData/VirgoOnline/SNEB_dbox_rack.cfg

For a 0.1V offset in these configurtation files we observe the following temperature changes in the sensor outputs:

The final offsets used in cfg files are then:

Since the re-opening of the IB-PR valve and the lock of the IMC, the temperature of the SIB1 bench started to increase. This is a known effect which is due to the warming of the optics by the main beam and, when the PR is parked in a "bad position" (see for exemple here and here), also to the back reflected beam from the PR, that stops on some optics/bench part instead of an apposite beam dump.

After few months of non-lock, the position of the PR could have drifted. With Paolo, we tried to bring back the PR close to its last good position and look at the SIB2_B2 photodiode to asses the aligned position (see fig. 1). We found out that the vertical position was quite off, but even after replacing the PR_Y DoF we could see any signal on B2.

However, looking at the MIR and MAR DoFs of the PR (see fig. 1), we see that for the same MAR_TX position, MIR_TX was off by ~ 150urad. This is probably due to the PR which slid on the wires during some of the test done in May. The back reflected beam was then touching the surface of the bench its, rising up its temperature.

Once found the aligned position, we left the PR with the usual misalignment and stored its new position on the PR Metatron node.

Starting from Saturday 11 july, the measured amplitudes of the 6 MHz and 56 MHz lines became noisy for an unknown reason. The 2 signals became noisy at the same time so we suspected a problem with the LNFS box that generates the two lines, while the 22MHz, generated on a different synthesizer (LNFS_2) doesn't show the same problem. 


In order to test this hypothesis, on July 15 at 9:00 UTC, the two LNFS boxes were exchanged, and the rms on the measured signal amplitude returned to normal. The rms of the 22MHz signal amplitude remained unchanged, and did not increase after the intervention.


At this point, we don't know exactly what causes the signals to become noisy.  A similar phenomenon has been already observed in the past, when the problem was associated to a temperature problem in the INJ IER. The temperature in IER rose by a few degrees and became more unstable since last month. This might be an explaination for the problem even if there is not a direct correlation observed at the moment.
 

The two LNFS boxes may be exchanged again in the future or swapped with one spare.
 

Since few week ago, after the intervention on INJ/ALS, the IMC longitudinal error signal shows a quite big offset (~ -1.8V), compared to the past( {-0.25, 0.1} ), visible as frequency to amplitude coupling on the INJ_PMC_TRA_DC_FMOD_COUPLING signal.

On Monday, we started to investigate the reason why this offset appeared.
We first checked whether the choise of the  22Mhz frequency modulation, used to lock IMC cavity, could itself be the cause of the offset. As we see in fig. 1, we started by putting at zero the IMC_OFFSET and scanning the 22Mhz on the LNFS up to few kHz of frequency shift. This resulted only in very small variations of the coupling channel.

Thus, we moved to investigate the way the beam enter the IMC cavity by acting on the BPC (with IMC_ON on 0, i.e. the BPC using its DCQPDs). As a result (fig. 2), the changes on the beam alignment improved the inejction into the RFC (visible on RFC_TRA signal), without changing the offset on the IMC longitudinal error signal.

Since July 8, 2026, the WEB external humidity and temperature sensors of the wheater station are producing wrong data.
First we did a restart of the VPM process BcMeteoWe (Detector s Environment Monitoring > Environment > BcMeteoWe) but it did not solve the issue. Then, today we went to WEB to check the console of the station, we looked at the cabling, disconnecting and reconnecting them, we also restarted the module. None of theese actions were usefull to restart the acquisition and no values of external temperature and humidity are read by the device, external pressure and wind parameters are correctly collected.

An inspection of the sensor placed on top of the building must be scheduled.
 

The swap to the emergency generator has been completed and a final test  with a sequence ~30 sec ENEL interruption -> generator intervention -> ENEL back -> generator offload has been satisfactorily performed.

Electrical situation is then back to default one.

A longer term solution for having the 2nd generator (GE 2) being able to serve both "UPS Central Building + UPS Computing Room + IPS  Central Building UTAs" and "IPS Central Building Chillers" is currently being planned (this solution is basically the first part of the generators parallelization that was already foreseen). 

ITF found DOWN and in UPGRADING Mode.
All times are UTC.
Below the list of activities comunicated to the Control Room:
07:41 - 10:03 Works in TCS Room (Ciardelli).
07:48 - 11:06 PAY IN installation in vacuum chamber (Majorana, Puppo, Pinto, and Benedetti, #69345).
07:54 - 08:40 NI PAY & tower electrical checks (Gherardini, Ricci, and Zaza in the tower, while Dattilo outside the tower, #69344, #69347, #69348). Tower electrical checks successfull.
The test of the backup generator has been performed twice in the afternoon (14:06, and 15:30), the IPS is now stable again and no other tests involving the shutting off of the IPS are needed. The setup of the Technical Building is back at the standard configuration using another generator #69351.
The activity in the CEB Clean Room is proceeding. 
 

Sub-system reports
SBE
SQB1: Recovered Vertical Position with Motor 3 rotating left, and Motor 4 rotating right.
DMS
TCS > Chillers: Shelved all DMS flags until the TCS AUX Lasers Cooling System will be installed.

Activity concluded at 11:06 UTC.