The new system is now working steadily, at around 1.8 (NI) - 2 (WI) liters per minute.
Before shutting the old system off, issues and anomalies with the flows (both NI and WI) were observed.
This phenomenon can be a consequence of the following issues:
Clogging in the piping
Clogging in the fittings
Clogging in the utilities
Old or damaged flowmeters
Presence of air
Insufficient circulators
Losses in the circuit
The new cooling system was designed specifically to comply to the correct water treatment and choice of materials to avoid future corrosion, deposition, erosion, pitting or biologic contamination and the flowmeters already in place were replaced. Air was completely removed and the circulator's power was doubled (from a single one for two branches to two separate ones).
Still, the measured flow of the new system was the same or lower than the old one, even when switching back to the old flowmeters, that show the same measurements as the new ones.
First thing that needs to be taken into account is that during the installation the system remained filled with still water for more than a month, so if any debris were present they might have deposited.
This is why Nicola, Piernicola and I carried out a thorough cleaning of every piece of piping and every fitting, laser area included, with air, mechanical friction, alcaline soap and both alcohol and ultrapure water ultrasound bath.
We inverted the laser's inlet and outlet from the circulators onwards and closed the system on itself at every level. The results are the following:
The flow up to the split among the utilities, under the bench, is 10 l/min.
After the splitting into 8 utilities, it decreases to 2 l/min (acceptable according to the laser manufacturer)
We tested every utility and found some beam dump to be completely clogged so that not even air could go through. The laser, nevertheless, after being cleaned with dust-free cloths and distilled watere, showed no working issues.
While dismantling and cleaning the old system, debris were collected and biological contamination was observed. They will be sent to an external laboratory to be analized.
Water pH was between 6 and 6.5.
TDS document addressing pollution in cooling systems coming soon
From a quick look:
The excess noise above 2kHz engaged as well on May 7th: as shown by VIM in Figure 1
This noise seems associated to the INJ HVAC: as it went off and on at the time of its switching off/on on July 28th : Figure 2.
This morning, according to the planned schedule, we attempted to switch on the NI and WI main CO₂ lasers.
After Nicola and Cecilia successfully restored the chillers and the Guardian system, the NI and WI main lasers were switched on at 08:17 UTC and 08:18 UTC, respectively.
Unfortunately, after about 1 minute and 40 seconds of operation, the WI main laser unexpectedly switched off (fig.1). Several attempts were then made to restart it, but without success.
At that point, Roberto and Flavio joined the investigation, and a series of checks was performed to determine whether the problem was related to the laser itself or to the driving electronics.
A visual inspection of the bench, including the BNC connections and the laser itself, did not reveal any obvious anomaly. The laser status LED was correctly lit in red, and all RF driver dongles were green, as expected.
After closing the bench, we performed one final startup attempt. This time the WI laser switched on, but only at approximately half of its nominal power. After another power cycle, the laser briefly reached its nominal output power before dropping again to approximately half power (fig.2).
During the brief period in which the laser reached nominal power, I acquired a thermal camera image of the DAS beam (fig.3). No anomalies were observed.
These observations suggest that the problem is more likely related to the electronics than to the laser itself. To be understood.For safety reasons, we decided to switch off both lasers on the WI bench.
At the time of writing, the laser status is as follows:
The WI main chiller is off. The backup chiller is connected only to the NI.
For the auxiliary chiller, the valve to the WI branch has been closed to avoid circulating cold water through the switched-off WI laser.
================================
As a side note (but an important one), I am experiencing significant difficulties with online monitoring using Data Display, even when using the previous version (Dy/v11r2/Linux-x86_64-AL9). While monitoring the channels, the displayed data frequently disappear for several seconds before reappearing. After some time, the application freezes completely and has to be restarted (see Fig. 4 for an example).
Acoustic noise inside the INJ room has changed significantly between the most recent switch off (July 28, https://logbook.virgo-gw.eu/virgo/?r=69458) and the previous which was in November (Nov. 27 2025: https://logbook.virgo-gw.eu/virgo/?r=68269). The changes are shown in the LAST plot attached comparing mics ASD. We tried to track these changes and found the following:
Reduced 20Hz acoustic bump:
On June 18th the RETURN fan frequency of INJ HVAC was changed from the usual 21 Hz to 14 Hz, and the acoustic bump around 20 Hz reduced significantly: Figure 1 and Figure 2. This is consistent with (but more pronounced then) what observed in our quick slow down in https://logbook.virgo-gw.eu/virgo/?r=67829 (return fan moved only down to 17Hz). We do not know why this reduction to 14 Hz was done, but the HVAC is running in this way since then. Good news is also that the environmental parameters seems not significantly affected (Figure 3).
Acoustic bumps 40-120 Hz changing level:
Examing VIM spectrogram back since Dec 2025, we noticed that in a number of occasions (see Table) some acoustic bumps in the INJ room mics increased or reduced. Bumps are roughly at 40, 60, 80 120 Hz. Figures 4 to 8 show some of these occurrences. The last increase happened on May 7th, and the noise condition is this since then. It looks that these occurences coincide with people entering and then leaving the INJ area (seismic activity). Examining some occurences, we often find a change in the pressure of the INJ room, yet no change in the fans' frequency: see in Figure 9 and following.
Excess noise around 490 Hz and above 2kHz: this has to be investigated .... :-)
A list of these occurences:
| UTC | |
|---|---|
| Dec. 13 around 12:00 | increase |
| Dec 15 around 8:00 | further increase |
| Dec 30 around 10:00 | decrease |
| Feb 2 around 9:00 | increase |
| Fec 14 around 7:00 | decrease |
| March 5 between 10:00 and 12:00 | increase |
| March 10 around 10:00 | increase |
| March 24 around 9:00 | decrease |
| May 4 around 7:00 | increase |
| May 7 starting from 14:00 (after long access) | further increase |
From a quick look:
The excess noise above 2kHz engaged as well on May 7th: as shown by VIM in Figure 1
This noise seems associated to the INJ HVAC: as it went off and on at the time of its switching off/on on July 28th : Figure 2.
ITF in COMMISSIONING, DOWN
Activities reported to the control room:
The assembly of the new cooling system for the TCS auxiliary lasers is complete.
It is a two-stage system (pic 1 - schematics)
1. Primary stage –. The tank temperature is maintained by one of two chillers, operating in a redundant configuration: one unit designated as the primary chiller (working) and a second unit
as a (non operating until needed) backup, to ensure continuous operation. In case of failure, the operator on shift will only have to switch the piping from one chiller to the other (pic 2: AuxCooling Chiller)
2. Secondary stage – From the tank (pic 3 - AuxCooling Tank), water is distributed through two separate piping and valve systems, one per laser (pic 4 (temporary) - Distribution and post-heating system assembly phase). Each system includes four pipe collar heaters, enabling post-heating control of the oscilations in temperature, to keep them within the 0.1 limit (more details in future entries that will be posted as comments of this one).
New sensors were installed and new channels were created, soon to be implemented on DMS. Together with the old ones, the system can be monitored using (see schematics):
The new system is now working steadily, at around 1.8 (NI) - 2 (WI) liters per minute.
Before shutting the old system off, issues and anomalies with the flows (both NI and WI) were observed.
This phenomenon can be a consequence of the following issues:
Clogging in the piping
Clogging in the fittings
Clogging in the utilities
Old or damaged flowmeters
Presence of air
Insufficient circulators
Losses in the circuit
The new cooling system was designed specifically to comply to the correct water treatment and choice of materials to avoid future corrosion, deposition, erosion, pitting or biologic contamination and the flowmeters already in place were replaced. Air was completely removed and the circulator's power was doubled (from a single one for two branches to two separate ones).
Still, the measured flow of the new system was the same or lower than the old one, even when switching back to the old flowmeters, that show the same measurements as the new ones.
First thing that needs to be taken into account is that during the installation the system remained filled with still water for more than a month, so if any debris were present they might have deposited.
This is why Nicola, Piernicola and I carried out a thorough cleaning of every piece of piping and every fitting, laser area included, with air, mechanical friction, alcaline soap and both alcohol and ultrapure water ultrasound bath.
We inverted the laser's inlet and outlet from the circulators onwards and closed the system on itself at every level. The results are the following:
The flow up to the split among the utilities, under the bench, is 10 l/min.
After the splitting into 8 utilities, it decreases to 2 l/min (acceptable according to the laser manufacturer)
We tested every utility and found some beam dump to be completely clogged so that not even air could go through. The laser, nevertheless, after being cleaned with dust-free cloths and distilled watere, showed no working issues.
While dismantling and cleaning the old system, debris were collected and biological contamination was observed. They will be sent to an external laboratory to be analized.
Water pH was between 6 and 6.5.
TDS document addressing pollution in cooling systems coming soon
We commissioned the WI instrumented baffle for the prealignment of the WE, repeating the procedure used yesterday for the NI (69514)
The starting position for the WE is (TX; TY) = (24.5; 136.8) urad
We then swept the NE right by about 100 - 110 urad until we intersected the symmetrical column c5 and repeated the process. Overall, the power detected were:
The median point identified by these positions is (<TX>;<TY>) = (24.5; 136.9) urad.
We repeated the process with a vertical scan:
The median point identified by these positions is (<TX>;<TY>) = (25.3; 136.5) urad.
We were able to apply the procedure much faster, as we become more familiar with the behaviour of the baffle, and the obtained accuracy is still within a few urads from the well aligned position of the WE.
This afternoon, both CH CO₂ lasers were successfully switched on using the new cooling system designed and installed by the EGO team.
Before switching on the lasers, Piernicola performed a complete check of both CO₂ benches. In particular, he reset the zero level of all power meters and verified the correct operation of all remotely controlled flip mirrors.
The lasers were then switched on sequentially:
For both lasers, the pulse width was set to 160 µs. For reference, the maximum output power is achieved with a pulse width of 199 µs.
The laser behaviours observed so far are shown in Fig. 1.
The lasers will remain ON overnight. According to the current plan, the main CO₂ lasers will be switched on tomorrow morning.
I also performed a quick check using the thermal cameras on both benches and acquired one thermal image of each CH beam. A comparison with the images acquired on 2026-04-07 did not reveal any significant changes in the beam propagation. The comparison is shown in Fig. 2.
Side note: while trying to copy the thermal camera images from PCINFRACAM to the control machines, I noticed that the procedure was no longer working. I therefore contacted the Computing Team, and Elian found that the old SSH Secure Shell client is no longer supported following the recent Linux upgrade. As a temporary workaround, Elian identified an alternative procedure and shared it with all operators to be used during the upcoming Tuesday maintenance.
ITF found in COMMISSIONING Mode and LOCKED_ARMS_IR State.
All times are UTC.
06:38 - 06:49 Recovered SBE_SPRB vertical position with loop closed (operator).
07:10 - 10:30 Instrumented baffle commissioning (Vallejo, Boldrini. #69514).
07:41 - 11:00 Activity on TCS AUX Cooling System (Dattilo, Cavalieri).
11:39 - 12:35 Safety inspection at MCB and WEB (Fabozzi, external personnel).
12:07 - 15:07 Activity on TCS AUX Cooling System (Dattilo, Menzione, Cavalieri).
13:20 - 14:00 TCS CH Lasers ON and Flip Mirrors checked (Spinicelli, Nardecchia from remote).
13:20 WE PCal laser turned ON (Rolland from remote, #69513).
13:31 - ONGOING Suspensions tuning (Ruggi, Pinto).
ITF left in DOWN State and in COMMISSIONING Mode.
We attempted to evaluate how the instrumented baffle could be used to prealign the end mirrors of the arm cavities.
Starting from the locked arms, we took references of the alignment condition of the test masses, of the BS and of the transversal position of the PR (Figs.1,2), we then unlocked the interferometer and misaligned both ITMs by 60 urad on TY to simulate the stage of the recovery when the beam has been centered on the ETMs using the markers by moving the PR and the BS, but the alignment of the test masses themselves is still unknown.
The starting position for the NE is (TX; TY) = (-107.8; 45.4) urad
Once we did that, we started sweeping the NE alignment until the sensors on the NI instrumented baffle detected power. We then refined the alignment of the NE until we maximized the power detected by sensors 0,1 and 2 of the column c17, in the left sector of the baffle. The goal was to use the detected power to attempt to center the beam on the middle sensor, equalizing the power on sensors 0 and 2, assuming that they correspond to the tails of the Gaussian profile of the beam. We achieved this result on the c17 column with intervals of 0.2 - 0.4 urad for the NE, notice that the procedure is cumbersome, because the detected power fluctuates significantly so one needs to be patient and consider an average over a minute or so.
We then swept the NE right by about 100 - 110 urad until we intersected the symmetrical column c5 and repeated the process. Overall, the power detected were:
The median point identified by these positions is (<TX>;<TY>) = (-105.53; 44.56) urad, that is remarkably close to the initial position.
We repeated the process with a vertical scan:
The median point identified by these positions is (<TX>;<TY>) = (-105.68; 43.96) urad, which is also close to the initial alignment of the NE.
Our conclusion is that the instrumented baffle is accurate enough to provide a prealignment of the beam that centers it on the NI withing a few urads from the correct alignment, so it seems a promising tool to be used in future recoveries.
We tried to repeat the process for the WI instrumented baffle, but we could not finish the procedure in the remaining time of our shift. Another commissioning windows has been allocated for this tomorrow morning.
WE PCal switch on at 13h20 UTC.
Yesterday afternoon, we performed a measurement of the current input beam mode mismatch toward the arms, with the CO2 lasers off (only PR Chrocc and SR RH are at nominal value).
Since the interferometer is not actually fully locked, we decided to use the green beam scan of the arms to assess the cold mistmach.
The scan started at 14.05UTC of the August 4th.
The results are shown in fig. 1, with both arms around ~1.5% of mistmatch.
N_matching = 1.6e-02
W_matching = 1.3e-02
The following report summarizes the measurement and analysis of scans performed on 03-08-2026 in order to measure the arm Gouy phase. The aim is to verify that we recover the same Gouy phase as it was during the end of the last observation run. The Gouy phase is measured from the mode spacing measured from the frequency scans. The procedure to perform the slow FSR scan can be found on this page https://wiki.virgo-gw.eu/Commissioning/OptChar/Opt_Char_shift
Measurements:
At 14:54 UTC, ITF is locked using the state LOCKED_ARMS_BEATING
At 14:55 UTC, 1 iteration of scan is run as a test to verify that everything work
At 15:00 UTC, 15 iterations of scan, ended at 15:33 UTC
Results:
Tables 1 and 2 compare the mode spacing of the first four higher-order modes during the end O4 run and in the scan measured on 2026-08-03 in the west and north cavity, respectively.
Table 1 : North Cavity
| HOM 1 | HOM 2 | HOM 3 | HOM 4 | |
| 02 December 2025 | 0.1035±0.0003 | 0.2092±0.0003 | 0.3191±0.0002 | 0.4276±0.0002 |
| 03 August 2026 | 0.1057±0.0003 | 0.2113±0.0002 | 0.3215±0.0002 | 0.429±0.002 |
The measured mode spacing is very similar in each scan but slightly larger in the recent scans.
Table 2 : West Cavity
| HOM 1 | HOM 2 | HOM 3 | HOM 4 | |
| 02 December 2025 | 0.10246±0.0007 | 0.2076±0.0002 | 0.3201±0.0002 | 0.4304±0.0005 |
| 03 August 2026 | 0.103±0.001 | 0.2085±0.0008 | 0.3195±0.0003 | 0.4290±0.0003 |
The measured mode spacing is very similar in each scan, larger in the recent scan for the first two higher-order modes and smaller for HOM 3 and HOM 4. A supposition to explain the discrepancy between the different higher-order modes could be the change of the input test masses, resulting in a different transverse RoC asymmetry. The higher-order modes, presenting a larger area, experience a different portion of the mirror surface and as a result experience a different effective RoC. This hypothesis could explain why for HOM 1 and HOM 2 the mode spacing is larger during the scan performed on 03 August 2026 and smaller for HOM 3 and HOM 4.
To visualize what the observed difference represents in the scan, Figure 1 and Figure 2 show two scans plotted together. The blue curve represents the median scan performed on 03 August 2026 and the orange curve the old scan performed on 02 December 2025. The four plots below the full scan are a zoom on the first four higher order modes. These plots show that the difference is actually very small, suggesting that the Gouy phase is almost the same as in December.
ITF found in DOWN with the automation requesting to lock in BEATING_DRMI_1F.
The DRMI_LOCK node was also stuck trying to reach the requested state MISALIGNED_PR_SR; to unblock the situation I restarted the nodes.
After that I performed a manual prealignment of WI mirror and I relocked the cavity.
At 6:00 UTC started the planned maintenance, below the list of the activity communicated in control room:
All the maintenance activities concluded around 10:30 UTC.
In the afternoon the INJ team worked input beam mode matching measurement, activity in progress.
Other
M 4.3 in Pisa; the earthquake opened the PR ID loop and all the SBE loops. Properly closed.
Over the last days, different VMs (ctrlX and farmnX) intermittently became unresponsive, leaving users temporarily unable to log in.
The cause was memory exhaustion: interactive user sessions were consuming nearly all of a machine's memory, starving the essential system services that handle authentication and access.
To prevent this, we have applied a memory limit to the aggregate of all user sessions on each machine, so that a share of memory is always reserved for essential system services.
It is a soft limit: when sessions approach the threshold the system gradually slows them down and reclaims memory. it does not terminate processes or interrupt anyone's work.
Under heavy load, large sessions may run somewhat slower; this is the intended trade-off to keep the machine reachable for everyone.
VMs should now stay responsive and authentication reliable even under heavy memory usage. Please report any unexpected behaviour.
The FdWRawBack server configuration has been updated to use the 117TB of disks available for the RAW_BCK online stream on the stol02 host
Operation perfromed at 2026-08-04 08h31m44 UTC
Yesterday, I put the sensors back in place and restarted the data acquisition
Yesterday, I put the sensors back in place and restarted the data acquisition.
Today we continued the work on the recovery of the lock. Just to put some context also here, during the limited time of these couple of weeks we are relocking on purpose without any TCS actuators, because of commissioning plan and forthcoming hardware interventions. The purpose is to restore things as much as possible, given that we've not been locking for four months, and we ancitipate not being able to go much farther than the beginning of the CARM offset reduction part of the lock acquisition.
We proceeded with the lock of the ALS system, that started already on Friday; today, thanks also to the increased stability of the WI actuation, we could move from the standalone arms to the beating signals and close the CARM and DARM control loops (same controllers as before, new improved ones that can profit of the ALS recent improvements will be developed later).
Then we moved to the CARM offset step, and then the realignment and lock of the DRMI:
We spent some time to check gains and phases (given that the optical configuration is different w.r.t. April): we did not find anything out of the ordinary with the exception of SRCL, which has half the gain of before; we then moved to the DRMI with 3f signals, and we could do the handoff with no problems. This configuration has a bigger deviation from the past: the 169MHz demodulation phase for MICH/SRCL has changed, and the gains of both loops needed some tuning. Nothing relevant for the PRC instead.
We took the occasion also to close the floating setpoints servo of the SDB1 local controls.
We saved the new parameters of the lock acquistion (labeled in the .ini files) and we left the DRMI locked with 3f signals.
The following report has been submitted to the On-call interface.
On-call events -> Air Conditioning
Title: Water Leak from the Autoclave System
Author(s): andreazzoli
| Called at: 21:00, 03-08-2026, by: Other colleague |
| Remote intervention: Started: ; Ended: |
| On-site intervention: Started: 21:22, 03-08-2026; Ended: 21:55, 03-08-2026 |
| Status: Resolved |
| Operator when issue resolved: None |
Details:
Following a report from the RSPP regarding a water leak from the pressure tank system, I conducted an inspection.
The water leak was attributable to the operation of the water treatment system. The system was left in normal operation.
* Note that any files attached to this report are available in the On-call interface.
The swap of the two LNFS done on July 15th has been done without exchanging the eth address of the devices. As a consequence, while restarting the automation, it was trying to communicate with the old address, thus inverting the {6, 8, 56 }MHz signals with the {22, 81}MHz. We stucked at the Fmoderr state of the INJ node, untill we swapped the ethernet cables of the two LNFS (to be noted: it seems there is the possibility to swap the ports directly into the configuration file of the LNFS server.
Because of this, we found a peculiar corner case of the FmodErr loop: after the exchange of address, the LNFS_FREQ_3 was not anymore setted on 56MHz (was instead the old 81). The check of the Fmod, however, is based on a 56MHz demodulated channel, that gave 0 correction to the loop. Eventually, no change (both MC_Z or LNFS) has been requested, and the 8Mhz frequency has not been changed (and so the 6 and 56, which are automatically updated once the 8 changes).
As a result, this morning we didn't have any modulation at 56MHz. The problem has been solved applied a reset of the 8MHz.
The Fmod loop will be updated consequently.
In order to better center the corrections for the Etalon control in the actuator dynamics we have changed in a 4 days ramp
NI to 20.1 and WI to 19.6