While performing these tests, we noticed that the SL crystal temperatures were unusually high.
By checking the trends, we observed that the temperatures started to rise and eventually reached approximately 70°C after the pumping diodes were switched off on 07/05.
This suggests that the temperature controllers are not responding properly in the present configuration.

Under normal operation, both the pumping diodes and the crystals have their temperatures stabilized by the Peltier modules (W signals on the fig attached) in order to maintain the temperatures at their setpoints (SET signals on the fig attached).The hot side of the Peltier modules is itself cooled by the chiller.

Usually, when the pumping diodes and the chiller are switched off, the correction signals go to 100 and everything remains at low temperature.
We had never paid particular attention to this behavior before, but it is already somewhat anomalous, since the controllers should regulate around the setpoints rather than continuously applying maximum corrections.

When we switched off the chiller and the pumping diodes on 08/05, the behavior was different for the crystals: instead of going to 100 as observed for the pumping diodes, the Peltier correction signals for the crystals dropped to 0, and the crystal temperatures started to increase (figure 1)
This behavior is currently not understood. 0 corrections should be the lowest temperature...

The crystals eventually reached approximately 70°C.
When restarting the SL, we will need to assess whether this may have caused any damage and whether the alignment has been affected.

After noticing the issue and we saw that the crystal temperatures were increasing again (figure 2) : without changing anything on the chiller nor the peltier corrections, at 11:30 UTC the temperature of the crystals started to increase again from 40 degrees towards 70 degrees.
So we went to the chiller room to restart the chiller. The Peltier modules only resumed normal behavior once the temperature became close to the setpoint (figure 3).

We found Sa_WI crate off since its power supply was broken (probably due to aging). We replaced it with a spare. Top stage control is now working properly again. 

A selective switch-on test of the Laser system chillers was performed to identify their individual noise contribution to the laser benches.

The test involved the NEOVAN head chiller (Termotec), the main chiller (OMI), and the NEOVAN electronics chiller.

During the test, the AHU used for venting the NI tower remained ON, while the CEB hall and Laser lab AHUs were temporarily switched off and the supply/return air distribution valves together with the outdoor air valves were closed.

The detailed test sequence is reported in the attached file.

The test was completed with all chillers left OFF.

This morning we went in the Eeroom to check the INJ chillers.

- the MAIN CHILLER Omi (which is used for the SL diodes in EEroom, SL crystal in LL and Beam dumps IPC (EIB) + IMC REFL (EIB but not connected) + PMC REFL (LB) + B2 (SIB2) + IPC2 (SIB1 flux blocked)) when switched off was giving the error A05. We changed the filter and switched it on and it was working fine. We switched it off again.

- the NEOVAN HEAD CHILLER Termotek (used for the Neovan head). We switched it on with the circuit closed on itself and this time the pump started (we observed this behaviour also in the past, maybe due to the air which enters while changing the filter)

Yesterday we went to the Laser Lab so Marco could inspect the environment for security reasons following the near miss incident. For that, the EIB was blocked briefly (and Paul took some measuments of the BPC optics distance taking advantage of the window of opportunity) but no other action was taken. 

ITF found in UPGRADING Mode and DOWN State.
All times are UTC.
From 08:46 to 09:26 Melo, Lagabbe and Galimberti went in CEB > Laser Lab;
Activity concluded at 12:06, ITF left in  in UPGRADING Mode and DOWN State.

This morning I installed a new version (v3r3) of the Hameg package in VPM. This new release allows users to set the maximum output current generated by the device and to modify the output voltage using ramps (as requested by TCS group)

In the meantime, I added three new buttons to both the TCS_PAW_NI_PowerSupply and TCS_PAW_NI_PowerSupply processes:

• one to turn the device output on/off,
• one to set the maximum current,
• and one to start a voltage ramp on the output.

To simplify the user experience, I also added a few predefined values in the pop-up windows linked to these buttons in the VPM web interface

The software was tested using the "VPM test" environment and a spare Hameg 8041 device. As soon as possible, I will also perform quick tests in the TCS room.

At 2026-05-14-02h34m43-UTC, no more data from Sa_WI received*

• SUSP_Fb server log report

  • 2026-05-14-02h34m43-UTC>WARNING-[TolmFrameBuilder::ControlMerging] frame: 1462761300.800000000: merging is triggered on timeout (internal 1462761301.161215040 > max 1462761301.160000000)
    2026-05-14-02h34m43-UTC>WARNING-[TfbSubFrame::Fill] GPS: 1462761300.800000000 - Source Sa_WI: 1104/2000 missing packet(s), 0 data break(s) (missing data are at the end of the vector)
    2026-05-14-02h34m44-UTC>INFO...-[TfbSourceManager::Clean] source Sa_WI (62:v07) has stopped -> waiting for data...
    2026-05-14-02h34m44-UTC>INFO...-[TfbSourceManager::Reporting] Sources reporting: / nSrc: 57, missing(1): Sa_WI
    2026-05-14-02h34m44-UTC>INFO...-CfgReachState> Active(Active) Ok

After this event the FbmAlp  server complained about frames delivered too late from all the SUS Automation nodes 

• FbmAlp server log report

  • 2026-05-14-02h35m02-UTC>WARNING-FdFrMrgr: Could not wait longer for frame parts, output 1462761308.0, nSources=31 first:FbsAlp
    2026-05-14-02h35m02-UTC>WARNING-FdFrMrgr:1462761308.0 frames from SUS_BS start to be missing (isReady)
    2026-05-14-02h35m02-UTC>WARNING-FdFrMrgr:1462761308.0 frames from SUS_NI start to be missing (isReady)
    2026-05-14-02h35m02-UTC>WARNING-FdFrMrgr:1462761308.0 frames from SUS_NE start to be missing (isReady)
    2026-05-14-02h35m02-UTC>WARNING-FdFrMrgr:1462761308.0 frames from SUS_PR start to be missing (isReady)
    2026-05-14-02h35m02-UTC>WARNING-FdFrMrgr:1462761308.0 frames from SUS_WI start to be missing (isReady)
    2026-05-14-02h35m02-UTC>WARNING-FdFrMrgr:1462761308.0 frames from SUS_WE start to be missing (isReady)
    2026-05-14-02h35m02-UTC>WARNING-FdFrMrgr:1462761308.0 frames from SUS_BS start to be missing (out)
    2026-05-14-02h35m02-UTC>WARNING-FdFrMrgr:1462761308.0 frames from SUS_NI start to be missing (out)
    2026-05-14-02h35m02-UTC>WARNING-FdFrMrgr:1462761308.0 frames from SUS_NE start to be missing (out)
    2026-05-14-02h35m02-UTC>WARNING-FdFrMrgr:1462761308.0 frames from SUS_PR start to be missing (out)
    2026-05-14-02h35m02-UTC>WARNING-FdFrMrgr:1462761308.0 frames from SUS_WI start to be missing (out)
    2026-05-14-02h35m02-UTC>WARNING-FdFrMrgr:1462761308.0 frames from SUS_WE start to be missing (out)
    2026-05-14-02h35m03-UTC>INFO...-FdFrMrgrFeed: 1462761308.0 receiving again frames from SUS_NI
    2026-05-14-02h35m03-UTC>WARNING-FdFrMrgrFeed: reject too old frame (1462761308.0) from SUS_NI; last produced was 1462761317.0
    2026-05-14-02h35m03-UTC>INFO...-FdFrMrgrFeed: 1462761308.0 receiving again frames from SUS_WE
    2026-05-14-02h35m03-UTC>WARNING-FdFrMrgrFeed: reject too old frame (1462761308.0) from SUS_WE; last produced was 1462761317.0
    2026-05-14-02h35m03-UTC>INFO...-FdFrMrgrFeed: 1462761308.0 receiving again frames from SUS_BS

After some investigations, restarting the SatServer allows to recover the correct running conditions for the the FbmAlp server, and the SUS Automation nodes

• 2026-05-14-06h32m16-UTC     process:SatServer stopped
  2026-05-14-06h32m20-UTC     process:SatServer started

The Sa_WI data are still missing

ITF found in DOWN, UPGRADING
No commissioning activities communicated to the control room
The shifted was ended after the daily meeting due to the absence of further activities for the day

ITF found in UPGRADING Mode and DOWN State.
All times are UTC. 
06:09 MAINTENACE Mode set.
09:05 Re-engaged SPRB_SBE Tracking Global Control disabled on 6th May 2026.
09:11 Re-engaged SDB2_SBE Tracking Global Control disabled on 5th May 2026.
09:57 SDB2_SBE Firmware udate (Masserot). 
10:07 MAINTENANCE ended, ITF back in UPGRADING Mode.
10:07 Closed PR F7 and MAR loops and guardians.
ITF left in UPGRADING Mode and DOWN State with SDB2_SBE firmware update still ongonig.
No other commissioing activities were communicated to the control room.

ITF found in Upgrading mode and in Down; ISYS off for chiller malfunctioning.

No commissioning activities communicated in control room.

We went to EEroom to investigate the chiller, since we noticed that the CHILLER_PIPE_TE monitor suddenly increased a lot (see Fig. 1). Inside the chiller room, there was an allarm for one of the two chillers (the THERMOTEK chiller), so we filled it in with water (water flow was low) and swapped the filter that seemed a bit dirty, but it didn't restarted after this operation. After some trials we decided to use the spare chiller instead.

However, the OMI chiller wasn't working properly as well (linked to the DMS allarms) but we couldn't identify the problem. So, we decided to switch off everything for safety reasons for the weekend (Neovan amplifier and slave laser pump currents wer set to 0). 

Profiting of the measurements made today, we extracted the plants of the four DoFs needed to design the control filters. The basic idea was to measure the several pole-zero structures in the mechanical plants, to compensate them in the control filters, in order to flatten the mechanical responses and have for all the four degrees of freedom the same open-loop with UGF at 40Hz, controlled by an integrator and a double pole to roll-off the transfer functions (see fig.1) (same as it was done for the BPC controls).

For future reference, the measured and fitted plants for tilt X/Y and shift X/Y are reported in fig.s 2,3,4 and 5, respectively.

The control filter to reduce the excess of noise visible on the TILT X dof spectra has been updated but will be tested next time.

Activities carried out during the day:

Noise injections on BPL loop carried out by INJ team
PR suspension control test
Switch-off of Neovan and slave lasers and chillers (flags shelved from today 17 LT until Monday) due to malfunctioning

Today EIB started to oscillate very heavily. We opened the controls but the bench didn't close anymore, because the TY started drifting. So, we went to laser lab to investigate the problem, and fixing a bit the position of the water tubes we found again a good position and the bench could be closed finally.  

%%% noise injections on BPL %%%

We injected this noise BPL_noise_flt

ACL_FILTER_SET     "BPL_noise_flt"    1     1    20    20
ACL_FILTER_ZEROS   "BPL_noise_flt"      0.01      0
ACL_FILTER_POLES   "BPL_noise_flt"      0.5    0
ACL_FILTER_POLES   "BPL_noise_flt"      500    0.5

on the POST error signals. The error signals are filtered with the filter named BPL_PAOLO, which is taken from the BPC loop

ACL_FILTER_SET        "BPL_PAOLO"    1         -3.3     10        20    
ACL_FILTER_POLES    "BPL_PAOLO"    0        0
ACL_FILTER_POLES    "BPL_PAOLO"    200        0.7
ACL_FILTER_ZEROS    "BPL_PAOLO"    10        0

The problem with this filter is that we are reintroducing noise at 10-100 Hz ( fig 2. in the entry made yesterday #69090), so the aim was to design a more adapted filter.

The GPS are the following (each duration is 3 min):

tilt x 08:14:00 UTC (fig. 1)

tilt y 08:20:40 UTC

shift x 08:35:00 UTC (fig. 2)

shift y 08:42:00 UTC

The last plot shows the loop open and closed with the new filters designed by Manuel (in BPL filters: BPL_tiltx_flt, BPL_tility_flt, BPL_shiftx_flt, BPL_shifty_flt).

%%% clipping on the corrections %%%

The corrections sent to the DAC are now clipped  -9.9 to +9.9 V (since yesterday we observed a misbehaviour of the PZT at 10V )

Updating the SBE server configurations for the DAC1955 V3 firmware, we fpund strange Tolm packet setting for some Acl*s servers running of the SQB1 rtpc, more precisely for

• the SQB1_LC server where the  TOLM packets for the "SQB1_DAC_LC_LVDT_out_INT" and "SQB1_DAC_PSD" DAC1955  was not sent at the correct frequency
  • TOLM_OUTPUT_PACKET_SLICED    "SQB1_DAC_LC_LVDT_out_INT"    50000    DAC1955_FREQ  with    DAC1955_FREQ (= 100000)
  • instead of TOLM_OUTPUT_PACKET_SLICED    "SQB1_DAC_LC_LVDT_out_INT"    10000    DAC1955_FREQ  meaning that only 2 over the 10 packets were sent 
  • same thing for  TOLM_OUTPUT_PACKET_SLICED    "SQB1_DAC_PSD" 50000     DAC1955_FREQ 

We set the correct parameters but when we restarted the SQB1_LC server, we lost all the input Tolm packets . To recover the Tolm input packets, a reboot of the SQB1_rtpc was required .

We made several trials and at the end we set

• for the SQB1_LC server; 
  • this line for the  SQB1_DAC_LC_LVDT_out_INT Tolm packet in order to have all the DAC sample: TOLM_OUTPUT_PACKET_SLICED    "SQB1_DAC_LC_LVDT_out_INT"    LC_LOOP_FREQ    DAC1955_FREQ
  • this line for  the SQB1_DAC_PSD Tolm packet as the DAC channels are only constante value : TOLM_OUTPUT_PACKET_SLICED    "SQB1_DAC_PSD" 50000     DAC1955_FREQ as previously
• for the SQB1_FI server,  as we would like to have output Tolm data flow and it s only temperature control loops
  • this line for the SQB1_DAC_01_INT Tolm packet : TOLM_OUTPUT_PACKET_SLICED    "SQB1_DAC_01_INT"    50000    DAC1955_FREQ   instead of  TOLM_OUTPUT_PACKET_SLICED    "SQB1_DAC_01_INT"    LOOP_FREQ(10000)    DAC1955_FREQ previously

With these configurations,

• all the servers are correctly running
• the output Tolm data flow is the same as before the modifications, meaning that the use of DAC1955 v3 firmware will be helpful 
• the SBE and LC loops were successfully closed 

Operations performed between 2026-05-07-13h45m08-UTC  and  2026-05-07-14h47m25-UTC

• This morning, before starting to work on the BPL loop, we unblocked the beam transmitted from the PMC and we took the references of its position on the BPL QPDs (see fig 1).
• Then we plugged the PZT to the DAC in Eeroom. The PZT driver accepts -2V +12V, while the DAC dynamics is +-10V but due to the LEMO 3 PIN - BNC adaptation it is restricted to +-5V. On the PZT driver the DAC input is summed with a offset (from 0 to 10 V) that can be manually adjusted, and then amplified by slightly more than a factor 10, to -30 +130 V.  We manually set the offset to mid range (5V), so that we can correct around this value with the DAC.

lines 127-130 in  /virgoData/VirgoOnline/ISYS_EER_dac.cfg

ACL_DAC_CH    dac1955_EER_DAC2_ch03    1    DAC1955_FREQ    BPL_PZT_EIB_M1BH_CORR            0            1            "None"                        "ad1955-10v"    ""
ACL_DAC_CH    dac1955_EER_DAC2_ch04    1    DAC1955_FREQ    BPL_PZT_EIB_M1BV_CORR               0            1            "None"                        "ad1955-10v"    ""
ACL_DAC_CH    dac1955_EER_DAC2_ch05    1    DAC1955_FREQ    BPL_PZT_LB_M14H_CORR            0            1            "None"                        "ad1955-10v"    ""
ACL_DAC_CH    dac1955_EER_DAC2_ch06    1    DAC1955_FREQ    BPL_PZT_LB_M14V_CORR            0            1            "None"                        "ad1955-10v"

• We injected a 2Hz line on the PZTs during 90s. Here the gps for each actuator

gps = [1462194492 1462194603 1462194722 1462194871]; actuators = {'INJ_BPL_PZT_EIB_M1BH_CORR','INJ_BPL_PZT_EIB_M1BV_CORR', 'INJ_BPL_PZT_LB_M14H_CORR', 'INJ_BPL_PZT_LB_M14V_CORR'};

• The sensors are: {'INJ_BPL_QF_h_norm', 'INJ_BPL_QF_v_norm','INJ_BPL_QN_h_norm', 'INJ_BPL_QN_v_norm' };
  • QF mainly see the tilt (done with EIB M1B and also a bit with LB M14)
  • QN mainly see the shift (done with LB M14)
• We computed the driving matrix with the attached script and closed the loop. We started to try different filters (see 2nd plot). As a final test we removed the resistences at the output of the PZT driver, from around 15:58 UTC to 16:20 UTC (but there is a lot of noise on the signals, introduced by the amplifier itself, so we put it back).

To do next:

• to perform noise injections and design a better filter which does not introduce noise on QF h between 20 and 100 Hz
• check availability for a DAC +-20V
• add in the ISYS_BPL code:
  • a latch to keep the setpoint when the loop is open 
  • clip on the corrections to avoid too large beam excursion
  • trigger to open the loop if there is no signal

Yesterday, additional inspections (already reported in 69077) of the PAM viewports were carried out using a thermal camera:

• NI: big viewport — visible coating delamination (see Fig. 1)
• WI: small viewport — no visible coating delamination (inspected only from the outside see Fig. 2)