ITF found in LOW_NOISE_3_SQZ and SCIENCE mode.
It unlocked at 08:45 UTC. After two failed attempts, the ITF was back in SCIENCE mode at 11:19 UTC.
ITF found in LOW_NOISE_3_SQZ and SCIENCE mode.
It unlocked at 08:45 UTC. After two failed attempts, the ITF was back in SCIENCE mode at 11:19 UTC.
On Monday 11th we replaced the distribution box connected to NE accelerometers in order to exclude it from the possible sources of the glitches. Unfortunately the test was unsuccessful: the glitches reappeared after a few hours. In the next days we will replace the boards.
On October 30, we completed the 19th height monitoring campaign of the West Tunnel - West End Building reference points.
Tab.1 summarizes the result of the survey and Fig.1 shows the longitudinal profile, that reports graphically the trend of the subsidence over time.
The automatic engagement of locking reallocation to IP for PR and BS works fine. Fig 1 shows the evolution of PR CORR and BS CORR slow rms: both minimum level and peaks in windy conitions are reduced a lot.
In fig 2, a comparison of data for medium level wind speed condition is shown. The lowering of PR and BS MAR correction correspond to a certain reduction of BS TY low frequency spectral level, but the gain in CMRF stability is quite poor.
In fig 3, the evolution of BS TY slow rms is added: the reduction of BS CORR corresponds to a visible (small) reduciton of BS TY. On the contrary, CMRF rms evolution seems not correlated.
Since the 2024-10-21-16h28-UTC , the FmRawBack server is stopped and none improvements were observed in the DAQ latency jumps
The FmRawBack server, running on the olserver119 and providing the raw_bck.ffl file, has been restarted at 2024-10-05-07h52m34-UTC.
I found ITF in SCIENCE; It remained locked for the whole shift.
NO DMS event.
Guard tour (UTC):
ITF found in LOW_NOISE_3 and in Science Mode.
At 16:56 UTC the ITF unlocked, at the same time the data of the ACS DET became unavailable. While investigating the issue I found the related UTA panel was switched off, and the DET humidier tank was spilling water. The water has passed throught a passing cable hole in the floor and hit the UPS in the room below, causing a switch off of the Air conditioniong of the DET Lab. D'Andrea was able to bypass the UPS, which now require repairment, and at around 17:55 UTC the UTA panel was restarted.
ITF back in LOW_NOISE_3 and Science Mode at 19:22 UTC. At 19:40 UTC, in agreement with Vardaro, I set the ITF back in LOW_NOISE_3_SQZ.
ITF left locked.
SBE
After the unlock at 16:56 UTC, I recovered the SPRB Vertical correction using the related Motor3.
Oncall events
Air Conditioning
(13-11-2024 18:10 - 13-11-2024 19:10) Operator on site with expert from remote
Status: Ended
Description: At 16:56 UTC, in parallel with the Unlock, the data from the ACS DET became unavailabe. After a brief investigation I found the the UTA DET was not reachable anymore. When I went to investigate the issue in the Central Building I've found the interface switched off, and the close tank of the humidifier was spilling water from the overflow valve.
Actions undertaken: I've contacted Soldani, D'Andrea find out that the water of the humidier tank has spilled out from the floor above and hit the UPS, triggering a switch off the the DET air condition (which could be the cause of the unlock). After the bypass of the UPS it was possible to restore the UTA Interface at around 18:00 UTC, and the system was restored.
Electricity
(13-11-2024 18:20 - 13-11-2024 19:10) On site
Status: Ended
Description: While investigating the issue with the UTA DET, I found that the related inferface was switched off. Soldani contacted D'Andrea, who went to the central Building to investigate the issue.
Actions undertaken: The DET ACS UPS was found wet due to the water spilling from the floor above and showing errors on screen, but still switched on. D'Andrea put the area in safety and bypassed the broken UPS in order to restore the ACS of the Detection Lab.
In order to reemove the lines I forced the switch off of the line in DOWN,
I re switch on the line in SQZ_INJECTING. I pressed load duirng the Troubleshooting
Yesterday morning, during the maintenance time, we made a visual inspection to the RFC tra setup. In particular, we tried to estimate which beam splitter is actually used to split the beam between RFC_TRA photodiode and camera without unmounting it. By only using an IR sensor card, we expect few mW of light going onto the camera, which would probably mean a 50/50 beam splitter or similar, and not the 1/99 as in the optocad design.
To be noted that, at the end of the intervention in LaserLab, we found that the RFC tra photodiode was almost completely misaligned. Since we didn't touch any optics, we couldn't identify which component moved. Anyway, the realignment procedure was quite difficult since the RFC doesn't lock without the transmission trigger, and the new photodiode surface is much smaller than the previous one.
I have compared the specturm of RFC Tra before and after replacement of the photodiode. To compare the spectra, I have used the spectrum of the new photodiode as it is, as it should be calibrated correctly into mW based on the value of the resistors used in the photodiode. For the old photodiode, I have adjusted the calibration to match the two spectra at 1501Hz, the frequency of the PSTAB line. They also matche well below 10Hz after the adjustments, which adds confidence that the comparison is done in the same units.
Figure 1 shows the two spectra. They agree well above 200Hz. Below that frequency there are bumps that are different frequencies, and the overall noise floor before the PD change seems lower. From a few mW on the photodiode one would expect a quantum shot noise of a few 1e-8 mW/rtHz, so at high frequency the spectrum is likely dominated by quantum noise. Depending on which data I look at I conclude the power is 2.3mW or 5mW on the PD. Hence it is difficult to be sure if it is only quantum noise.
At lower frequency one can try to explain the noise floor with a 1/f^0.66 shape. I honestly don't know if it is really the noise floor, or just the effect of the many bumps give a floor which looks to have the same slope as the mystery 1/f^{2/3} noise.
ITF found in Prepare_Science with Mattia working from remote trying to understand and fix the lcking troubles; in the meantime Paolo reached the site
At 6:26 UTC we reached LN3; we decided to set Adjusting mode to allow Paolo to recover the proper PR vertical position. To do that I went in CB to switch ON the PR motors; the ITF unlocked during this operation.
At 6:51 UTC ITF in Troubleshooting mode; Paolo was able to recover the PR and the BPC (at the end of this activity I switched off the PR motor drivers).
After one failed attempt we could relock the ITF in LN3 and Science mode was set at 9:19 UTC.
Then, at 10:21 UTC the ITF unlocked; after one failled attempt we could relock the ITF in LN3 and Science mode was set at 11:33 UTC.
At 12:56 UTC ITF in Adjusting mode for one minute to allow Marco to switch off the dithering lines of the Squeezing.
SBE
SPRB vertical actuator close to saturation; to restore at the next unlock.
I was called at about 5:30 local time. The ITF had unlock continously over the night.
There are at least 15 unlocks to look at, all apparently for different reasons. While I could not check all of them, I found an unlock during ACQUIRE_LOW_NOISE_3 apparently due to BS_TX fbw being engaged while a 400 mHz oscillation was affecting the loop (Fig.1). While I was looking at another unlock, the ITF managed to reach LN2 by pure chance, and I took the occasion to comment line 6137 in ITF_LOCK.py to prevent the engagement of BS_TX fbw.
This worked, the ITF reached LN3. At this point, I manually engaged the fbw configuration for BS_TX and restored ITF_LOCK.py as it was.
Unfortunately the ITF unlocked shortly after, due to a high gain oscillation of DIFFp_TY. The servos of DIFFp were not functioning due to low line coherence (Fig.2)
In the last attempt, I replicated the BS fbw trick and increased (online, without saving) the DIFFp lines amplitude as a patch (by about 60%). The Audio channels on B1 are not saturating as a consequence.
The servo started working again and changed DIFFp gains significantly. This solution seems to work and at the moment the lock has lasted about 10 minutes.
I am closing this intervention and moving on site to continue.
ITF was found in CARM_NULL_1F and PREPARE_SCIENCE mode.
Unfortunately, the increasing wind activity (fig1) prevented the lock for the first part of the shift. BAD_WEATHER mode set.
At 02:30 UTC, although the weather conditions improved, ITF unlocked however at various steps. TROUBLESHOOTING mode set.
I contacted the ISC OnCall to investigate it. Boldrini commented a line in ITF_LOCK.py which closes BS loop in full bandwidth.
ITF relocked at LN3 only at 05:32 UTC. SCIENCE Mode set at 05:34 UTC but ITF unlocked again at 05:43 UTC.
Relocking in progress...
Guard tour (UTC):
22:06 - 22:36
00:00 - 00:30
02:00 - 02:31
04:00 - 04:28
ISYS
BPC - BsX_TX started to drift this afternoon and reached a value of 14 (plot3). Experts informed, and in accordance with Melo, we decided to operate on it on the morning shift.
At 02:56 UTC the B2 QD2 galvo loop remained open. I closed it via VPM.
SUSP
Sa_PR_F0_COIL_V1_50Hz has drifted out of range since this afternoon (plot2). Experts informed, and in accordance with Ruggi, we decided to operate on it on the morning shift.
ISC
(13-11-2024 04:20 - ) From remote
Status: On-going
Description: Problems with the lock acquisition
ISYS
(13-11-2024 01:45 - 13-11-2024 02:00) From remote
Status: Ended
Description: BPC - BsX_TX drifting out of range
NO actions for this night
SUSP
(13-11-2024 00:15 - 13-11-2024 00:15) Other
Status: Ended
Description: Sa_PR_F0_COIL_V1_50Hz drift out of range
NO actions for this night
ITF status in BAD WEATHER, strong wind activity that affect the lock. 16:50 UTC ITF back in SCIENCE (LOW_NOISE_3). 18:16 UTC ITF in SCIENCE (LOW_NOISE_3_SQZ). 18:56 UTC SQZ was disengaged, and the ITF transitioned to LOW_NOISE_3 so I put SCIENCE again. 20:36 UTC ITF unlocked. ITF left in relocking acquisition.
Sub-system reportsSQZ
SQB2 SBE and LC control left open as requested by M. Vardaro
This morning, the inspection and increase of rodent traps were also carried out inside the experimental buildings.
The personnel of an external company, under the supervision of N. Baldocchi, increased the number of rodent traps installed in terminal buildings to mitigate the damage caused by them.
As part of the same intervention, an inspection was conducted in the central building to review traps installed in previous operations. During the inspection, any damaged devices were replaced.
The monitoring and implementation of the traps is ongoing, to ensure the effectiveness of the measures adopted and, if necessary, to plan additional actions.
ITF found in relocking and in PREPARE_SCIENCE Mode. It was not possible to relock due to the ITF unlocking at various steps of the locking acquisitions.
At 7:00 UTC I set the ITF in MAINTENANCE Mode and manually unlocked. Here a list of the activity reported to the Control Room:
- Cleaning of Scientifc Area (External firm managed by Ciardelli);
- OMC Scan/Lock (Operator);
- At 7:20 UTC, TCS Chiller Refill (Ciardelli);
- Cranes Inspection (Bazzi);
- At 8:05 UTC, Check of IMC Working Point and Position (Operator);
- TCS power adjustment (Nardecchia);
- From 8:20 UTC to 9:00 UTC, TCS Camera reference and Power checks (Operator);
- From 9:30 UTC to 11:10 UTC, INJ activity inside Laser LAb / EE Room (Spinicelli, Montanari, Melo, De Rossi);
- ACS check on DET Humidifer (external firm managed by Soldani);
- At 10:00 UTC, Final check performed by Nenci.
At 11:10 UTC, after the INJ activity was concluded, I started the relock.
In agreement with the expert, at 13:42 UTC we put the ITF in TROUBLESHOOTING and DOWN State to allow Montanari to go back to the NE to disconnect the accelerometers of the Suspension to revert the action performed yesterday.
ITF left unlocked.
DET
7:20 UTC - 5 minutes of OMC Lock in NI Single Bounce configuration
From 7:27 UTC to 7:47 UTC - OMC Scan
TCS Power Checks
CH [W] | INNER DAS [W] | OUTER DAS [W] | ||
W | on the pickoff | 0.28 | 0.05 | 0.258 |
N | on the pickoff | 0.646 | 0.047 | 0.602 |
This morning, the inspection and increase of rodent traps were also carried out inside the experimental buildings.
The personnel of an external company, under the supervision of N. Baldocchi, increased the number of rodent traps installed in terminal buildings to mitigate the damage caused by them.
As part of the same intervention, an inspection was conducted in the central building to review traps installed in previous operations. During the inspection, any damaged devices were replaced.
The monitoring and implementation of the traps is ongoing, to ensure the effectiveness of the measures adopted and, if necessary, to plan additional actions.
The NE and WE PCal config has been changed to allow more transparent switch of control loop photodiodes. When the switch button is pressed it will automatically change the photodiode use to create the hpcal channel used by hrec to do the line subtraction. The channel are now named V1:PCAL_*_PD_hpcal instead of V1:PCAL_*_PD1_hpcal.
For NE the control loop photodiode has been put back to Tx_PD1, as using Tx_PD2 was not solving the glitch issue.
Hrec and HrecN were restarded at 9h27 UTC.
ITF found in CARM_NULL_1F and in PREPARE_SCIENCE mode.
Back in LN3, SCIENCE at 22:40 UTC.
It unlocked at 23:46 and at 23:59 UTC; after the unlock the OMC shutter remained open (light on B1s2). To restore the situation I applied the procedure: NI single bounce then, from DET MAIN, ask for FORCE_CHECK_OPEN --> SHUTTER_OPEN --> SHUTTER CLOSED.
Back in LN3, SCIENCE at 02:13 UTC.
At 05:35 UTC I noticed an increasing of the ENV..LL_MAIN_CHILLER_PIPE_TE (fig3), I set ITF in TROUBLESHOOTING and I went in EE Room to check the status of the main laser chiller. OnCall expert told me to leave it as it is and wait for the maintenance intervention. Unfortunately my intervention unlocked ITF.
Relocking in progress, PRREPARE_SCIENCE mode set.
Guard tour (UTC):
22:20 - 22:54
00:00 - 00:33
02:00 - 02:33
04:00 - 04:49
DAQ
04:31 UTC - TelescreenArt provess crashed. Restarted
ISYS
ENV..LL_MAIN_CHILLER_PIPE_TE increased.
ISYS
(12-11-2024 05:35 - 12-11-2024 05:45) Operator on site with expert from remote
Status: Ended
Description: ENV..LL_MAIN_CHILLER_PIPE_TE increased.
I touched the probe and the temperature got stuck.
I found the ITF in SCIENCE mode at 15:21 UTC. The ITF was in the process of ADJUSTING for a distribution box upgrade at the NE accelerometer (Boschi, Montanari), with a test reallocation for PR BS (Ruggi, Bersanetti). The ITF was then unlocked at 15:31 UTC, and the status was in TROUBLESHOOTING. At 16:01 UTC, together with D. Bersanetti, we decided to start the unlocked CALIBRATION early.
Here is the summary:
At 20:35 UTC, the ITF returned to SCIENCE mode without SQZ (the SQZ does not work due to humidity problems; actions will be performed tomorrow during maintenance). At 20:56 UTC, the ITF was unlocked again. At 21:43 UTC, the ITF returned to SCIENCE mode, and at 21:56 UTC, the ITF was unlocked once more. The ITF is currently left in relocking acquisition.
Following the recent work on the improvement of the management of the locking correction in the suspension chain of PR and BS (entry 65424), we added two new Gnames in the SatServer configuration in order to turn on/off the new strategy during the lock acquisition. Such Gnames (for PR and BS) are called HCIP_ENBL and they are 0 by default, and 1 when the new strategy is in operation.
The new strategy has been tested while already locked (15:22:20 UTC), and the Gnames have been added to the Automation, in the ITF_LOCK node: flags are put to zero in DOWN, and to 1 at the end of ACQUIRE_LOW_NOISE_3, together with the turnoff of the DRMI lines.
As a reminder, given the "recent" almost TANGO-less implementation of the SatServer, the procedure for adding new Gnames is now the following:
GNAME = number
in the corresponding section (for this example, [Sc_BS]); the number is the one found above;Figure 1. After the squeezing was disengaged two bumps appeared in the specturm around 25Hz and 30Hz. These are related to the squeezer automatic alignment dither signals, and likely mean the residual B1p beam is misaligned on the squeezing benches when the squeezer is switched off.
ITF found in LOW_NOISE_3 and in Science Mode.
At 9:28 UTC I unlocked the ITF and set it in Troubleshooting Mode to allow the planned Replacement of the NE Distribution Box, carried out by Boschi and Montanari.
At 11:40 UTC we started to relock, ITF back in Science Mode and LOW_NOISE_3_SQZ at 12:47 UTC.
The ITF was set in Adjusting mode from 13:13 UTC to 13:18 UTC to allow Bersanetti to reload SusDAQBridge to add a new GNAME for the GIPC.
ITF left locked.
ACS
During the night the humidty inside the DET Lab decreased (See attached plot). During the morning Soldani increased the setpoint of the loop, and it was able to engage the Squeezing again. Further checks will be perfomed tomorrow during the maintenance.
Figure 1. After the squeezing was disengaged two bumps appeared in the specturm around 25Hz and 30Hz. These are related to the squeezer automatic alignment dither signals, and likely mean the residual B1p beam is misaligned on the squeezing benches when the squeezer is switched off.
In order to reemove the lines I forced the switch off of the line in DOWN,
I re switch on the line in SQZ_INJECTING. I pressed load duirng the Troubleshooting