Offline Hrec test setup

While installing an offline Hrec test setup in the CascinaTest Cm domain,  this afternoon I duplicated the Hrec configuration, updating the FD_IN parameters but forgotten to update the FDOUT ones .As consequence, when I started this offline server on the same host where the online Hrec server is running (olserver52), there were some troubles with the Hrec online  data .I didn’t realize it was creating problems as I was working with offline data and in the CascinaTest Cm domain. When I realized this , I contacted the Control room and I fxed the issue .

This issue started at 2024-04-26-14h08m02-UTC and was fixed at 2024-04-26-14h40m34-UTC

Looking at the ISC_Fb logfile, one can find  these 2 messages:

• 2024-04-26-02h22m33-UTC>WARNING-[TfbSubFrame::Fill] GPS: 1398133371.600000000 - Source Sc_PR: 22/2000 missing packet(s), 0 data break(s) (missing data are at the end of the vector)
  • It means that  that 22 Sc_PR Tolm packets over the 2000 expected for a frame at 5Hz are missing.
• 2024-04-26-02h22m33-UTC>ERROR..-[TfbSource::StoreData] producer Sc_PR: overflow at GPS 1398133371.999912050 (received 2021 / expected 2000) (1323 times)
  • this one is linked to each channel and it is a tagged message meaning that only the content of the last one is displayed with in parentheses the number of messages produced
  • the Sc_PR Tolm packet contains the data of 63 channels: 1323/63 = 21 Tolm packets
  • it means that 21 Sc_PR Tolm packets have been added to the 2000 expected

According to these 2 messages, as the frame building is done at 5Hz, it seems that

• at for the frame GPS 1398133371.600000000 + 0.2, 22 Sc_PR Tolm packets were missing
• while for the frame GPS 1398133371.800000000 + 0.2, 21 Tolm packets have been added to the 2000 expected

This plot at its zoom show the LSC_PR_Z_CORR and  Sc_PR_MIZ_LSC_CORR for  the 2024-04-26-02h22m33 event where

• the purple rectangle show the 22 missing Sc_PR Tolm packect starting at GPS1398133371.7978
• the orange rectangle show   the time periode from GPS1398133371.8 to GPS1398133371.8364 where Sc_PR Tolm packets are correctly received but the channels content are at zero at for the Sc_PR_MIZ_LSC_CORR  one

Related the 2024-04-13-19h35m32 Sc_PR event

ISC_Fb logfile report

• 2024-04-13-19h35m31-UTC>WARNING-[TfbSubFrame::Fill] GPS: 1397072149.600000000 - Source Sc_PR: 230/2000 missing packet(s), 0 data break(s) (missing data are at the end of the vector)
  • 230 Sc_PR Tolm packet missing for the 5Hz frame time stamped 1397072149.600000000
• 2024-04-13-19h35m31-UTC>ERROR..-[TfbSource::StoreData] producer Sc_PR: overflow at GPS 1397072149.999912000 (received 2230 / expected 2000) (14490 times)
  • 230 Sc_PR Tolm packets have been added to the 2000 expected for the last frame GPS 1397072149.8 + 0.2

This plot  show the LSC_PR_Z_CORR, Sc_PR_MIZ_LSC_CORR  and others Sc_PR channel for  this event where

• the purple rectangle show the 230 missing Sc_PR Tolm packect starting at GPS1397072149.7770
• the orange rectangle show   the time periode from  GPS1397072149.8 to  GPS1397072149. 8355 where Sc_PR Tolm packets are correctly received but the Sc_PR  channels content are at zeros

it occured a second time . VPM server restared at 2024-04-24-10h25m59-UTC

This morning, from 7h00 utc to 8h10 utc, we have added the channels slow_shutter_OMC1_T1_slow and slow_shutter_OMC1_T1_fast in the SDB1_OMC process. They will be used from now to monitor the state of the OMC slow shutter.

The channel slow_shutter_OMC1_T1_fast monitors the closing of the fast shutter (see Fig.1) which is presently done with the "MOVETOLIMIT" command at a speed of 1700 steps/s. The RMS of this channel is now used as observable in DetMoni to check if the shutter motor has stopped after the closing which is supposed to take place only when the ITF is DOWN in standard operations. Thus, if the ITF_LOCK index is above 1 and if the RMS of slow_shutter_OMC1_T1_fast indicates that the shutter is still running, a red flag will be triggered in the DMS.

The channel slow_shutter_OMC1_T1_slow monitors the opening of the fast shutter (see Fig.2) which is presently done with the "MOVEREL" command at a speed of  666 steps/s. The RMS of this channel is now used as observable in DetMoni to check if the shutter motor has stopped after the opening which is supposed to take place only at index 109 of ITF_LOCK. Thus, if the ITF_LOCK index is above 109 and if the RMS of slow_shutter_OMC1_T1_slow indicates that the shutter is still running, a red flag will be triggered in the DMS.

Since we had to start/stop the SDB1_OMC process several times for these operations, we verified at the end of the activity that the demodulation phase of the OMC locking signal was still correctly tuned (Fig.3).

Around  2024-04-20-06h06m28 most of the TolmFrameBuidler servers were unable to transmit theirs frames : it s probably the cause of the ITF unlock

Below the message reported by some TolmFRameBuilder servers

• ISC_Fb
  • 2024-04-20-06h06m28-UTC>ERROR..-frame 1397628405 not send to FbmFFE: sending queue is full
  • 2024-04-20-06h06m29-UTC>INFO...-output queue to FbmFFE has been flushed; Sending frame 1397628407
  • 2024-04-20-06h16m30-UTC>ERROR..-frame 1397629007 not send to FbmFFE: sending queue is full
  • 2024-04-20-06h16m31-UTC>INFO...-output queue to FbmFFE has been flushed; Sending frame 1397629009
• SWEB_Fb
  • 2024-04-20-06h06m28-UTC>ERROR..-frame 1397628405 not send to FbmFFE: sending queue is full
  • 2024-04-20-06h06m29-UTC>INFO...-Memory Used increase, cur. 756764.00(KB), inc. 40.00(KB)
  • 2024-04-20-06h06m30-UTC>INFO...-output queue to FbmFFE has been flushed; Sending frame 1397628408
  • 2024-04-20-06h16m30-UTC>ERROR..-frame 1397629007 not send to FbmFFE: sending queue is full
• SSFS_Fb
  • 2024-04-20-06h06m28-UTC>ERROR..-frame 1397628405 not send to FbmFFE: sending queue is full
  • 2024-04-20-06h06m29-UTC>INFO...-output queue to FbmFFE has been flushed; Sending frame 1397628407
  • 2024-04-20-06h16m30-UTC>ERROR..-frame 1397629007 not send to FbmFFE: sending queue is full
  • 2024-04-20-06h16m31-UTC>INFO...-output queue to FbmFFE has been flushed; Sending frame 1397629008
• SUSP_Fb
  • 2024-04-20-06h06m28-UTC>ERROR..-frame 1397628405 not send to FbmFFE: sending queue is full
  • 2024-04-20-06h06m31-UTC>INFO...-output queue to FbmFFE has been flushed; Sending frame 1397628409
  • 2024-04-20-06h16m30-UTC>ERROR..-frame 1397629007 not send to FbmFFE: sending queue is full
  • 2024-04-20-06h16m33-UTC>INFO...-output queue to FbmFFE has been flushed; Sending frame 1397629011

The IT departement has been informed to check for a possible network switch issue

We have updated the Shutter flag in the DMS (configuration of process DetMoni). In case the slow shutter motor remains stuck after step 109 of the lock acquisition, a red flag will be displayed.

We plan to further update the flag logic at the next maintenance to also perform some checks during the lock acquisition after the shutter closure.

This afternoon we performed a few tests on the OMC shutter with the ITF in NI single bounce in order to find a solution to the problem of OMC shutter getting stuck at the opening.

Fig.1 shows a test when sent a MOVETOLIMIT command via VPM to open the shutter. In this case the shutter motor get stuck, and we stopped it by sending the STOP MOTION command.

We have tested the opening of the shutter by sending some MOVEREL commands with a defined number of steps. We have found that with about 36000 steps the power on B1s reaches a plateau. Adding a bit of margin, we decided to perform 42000 steps to open the shutter. This is tested on Fig.2. We can see that it takes about 65 seconds to accomplish this motion.

We have then replaced the MOVETOLIMIT command by the MOVEREL command for the opening of the shutter in the class SHUTTER_OPENING of DET_MAIN. In the DET_MAIN.ini file, we have added the parameter nominal_open_steps = 42000, and we have updated the opening_duration to 80 seconds.

The opening and then closing of the shutter are tested with the automation on Fig.3. The closing is still performed with the MOVETOLIMIT command in order to be sure that we always start opening the shutter from the same position.

It was then possible to relock the ITF in Low Noise 3 at first trial.

The first lock at 16h00-UTC was luky as  the OMC lock noise disappeared just after the OMC lock with B1_PD3_Audio (plot) but not the others ones  (ie 21h50m-UTC one)

Here we report a few more observations concerning the excess noise causing problems to reach DC readout.

The excess noise is visible on the audio channels of several photodiodes (Fig.1 and Fig.2): B1s, B1sP and B1_PD3 (which are on the same service mezzanine) as well as on the B1p_PD1 audio channel (however in this later case, only lines are present, while for B1s and B1_PD3 there is a clear impact on the noise floor).

As reported already by Diego, an extra noise is also present on the SDB1_LC_TX_err and SDB1_LC_TY_err signals at the same time as the noise is present on the photodiodes. This SDB1_LC_TX/TY_err signals are constructed from the SDB1 B5 QD2 H and V signals, which also see the extra noise. This observation concerning the SDB1 local control was already true yesterday (Fig.3)

Both the spectrum of the B1_PD1 audio channel, and those of SDB1_B5_QD2_H/V signals are clean when we are in CARM_NULL_1F (Fig.4).

The extra lines on B1p_PD1 and the extra noise on the B5 QD2 signals show up as soon as the OMC shutter is open (Fig.5). The lines showing up on B1_PD1 are at the same frequency as the lines seen on B1s (Fig.6), although the B1s noise floor changes when the lock of the OMC is acquired.

The following plots shows a 300s FFTTime of the SDB2_B1p_PD1_Audio, SDB2_B1_PD3_Audio and the  SDB2_B1s_PD1_Audio channels during the OMC lock acquisition where the purple rectangle highlights  the periods with the presence of unexpected lines

• successfull OMC lock
  • 2024-04-01-00h45m00-UTC : FFTTime plot
  • 2024-04-04-13h34m00-UTC:  FFTTime plot,
    • this FFT plot compare the ASD  of the related channels (Audio and sample) for 2 periods: when some lines are present(purple)  and a quiet period(blue}
  • 2024-04-07-19h41m00-UTC:  FFTTime plot
  • 2024-04-09-20h43m00-UTC:  FFTTime plot
    • this FFT plot compare the ASD  of the related channels (Audio and sample) for 2 periods: when some lines are present(purple)  and a quiet period(blue}
  • 2024-04-14-10h52m00-UTC:  FFTTime plot :
    • During the OMC temperature scan, there is a period (blue rectangle) where the lines amplitude increases in the SDB2_B1s_PD1_Audio channels and appears in the  SDB2_B1_PD3_Audio channel
    • this FFT plot compare the ASD  of the related channels (Audio and sample) for 2 periods: when some lines are present(purple) and a quiet period(blue}
  • 2024-04-16-00h09m00-UTC: FFTTime plot
    • same statement for the blue rectangle as the previous event
  • 2024-04-17-03h40m40-UTC: FFTTime  plot
    • same statement for the blue rectangle as the previous event
  • From all these plots it seems that
    • there is a noisy time period (purple rectangle) when the OMC shutter is opened . This time period has increased with the time  and seems to start to reduce (see last event)
    • when the noisy time period is long, during the OMC temperature scan  a more noisy period occurs (blue rectangle

• unsuccessfull OMC lock
  • 2024-04-16-21h24m20-UTC: FFTTime plot
    • the noise does not disappear but seems to increase according to the OMC temperature analysis (blue rectangle) to remain when the OMC is locked
    • this FFT plot and its zoom compare the ASD  of the related channels (Audio and sample) for 2 periods: at the beginning when the OMC shutter is opened (purple) and when the OMC is locked with B1_PD3 (blue) : the noise has increased when the OMC is locked with B1_PD3
  • 2024-04-17-00h14m00-UTC: FFTTime plot
  • For reasons unknown today, the noise present when opening the OMC shutter does not disappear and moreover increases triggering the unlocking of the OMC

For debugging purposes, the channels Audio and DC related to the B1_PD3 and B1s_PD1 photodiodes acquired at 100KHz are now stored with the "_FS" suffix and available in the RAW_FULL stream .

• operations complete at 2024-04-17-15h25m26-UTC

Below the list of the channels, acquired by the SDB2_DBOX_RightDown - SDB2_MezzPD2  mezzanines at 100KHz and stored with the  "_FS" suffix:

 V1:SDB2_B1_PD3_Audio_raw_FS              nData= 100000 min= -6.11e+03  mean=-4.89e+03  max=-3.56e+03 rate= 100000Hz
 V1:SDB2_B1_PD3_DC_raw_FS                 nData= 100000 min= -3.55e+03  mean=-3.31e+03  max=-3.07e+03 rate= 100000Hz
 V1:SDB2_B1sP_PD1_Audio_raw_FS            nData= 100000 min= -2.69e+05  mean=-2.67e+05  max=-2.65e+05 rate= 100000Hz
 V1:SDB2_B1sP_PD1_DC_raw_FS               nData= 100000 min= -3.62e+03  mean=-3.38e+03  max=-3.14e+03 rate= 100000Hz
 V1:SDB2_B1s_PD1_Audio_raw_FS             nData= 100000 min=  6.12e+04  mean=  6.2e+04  max= 6.28e+04 rate= 100000Hz
 V1:SDB2_B1s_PD1_DC_raw_FS                nData= 100000 min=  1.54e+04  mean= 1.57e+04  max=  1.6e+04 rate= 100000Hz

Some plots showing the efficiency of the DARM_FREQ clipping in the case of  burst of glitches:

• events of the 2024-04-08 around 17h32m-UTC trend : burst of glitches on NARM and WARN
  • raw at 2024-04-08-17h32m20-UTC  and the zoom where the period with the clipping enabled is highlighted (red rectangle)
  • raw at 2024-04-09-17h37m-UTC
• events of the 2024-04-09 around 14h45-UTC trend:  burst of gliches on WARM only
  • raw  at 2024-04-09-15h48-UTC
• events of the  2024-04-14-23h20m-UTC trend: burst of glitches on NARM and WARM
  • raw at 2024-04-14-23h20-UTC

There is  several single glitches mainly of WARM like the following ones

• 2024-04-15-02h16m22-UTC : trend, raw
• 2024-04-14-22h25m19-UTC: trend, raw
• 2024-04-12-01h55m-UTC: trend, raw_01h57m14-UTC. raw_02h01m23_1s-UTC, raw_02h05m_80s-UTC

The issue with the extra spike at the beginning of the colored noise injection was related to time settings in the ACL commands in the ENVnoise process. These modifications were made by Alain (on March 26):

Figure 1 shows the result of the test: blue is with the OLD settings, purple is the with the NEW settings (the same colored noise with amplitude 0.025 V has been injected in both cases). The spike is no longer present.

The same test was done by injecting the colored noise from the ENVnoise and also using the ENV_MAIN node which calls the AcousticInjectionCEB process.

After the intervention of last Monday, the DIFFp_TX_SET servo was trying to build a signal with a line that did not exist anymore.

We changed the frequency of the demodulation to 3.3 Hz in ASC_pre and the frequency of the high-pass cut-off in DIFFp_alignment to 3.3 Hz as well. We also tuned the demodulation phase of DIFFp_TX_B1p to -1.8.

After that the offset manually added during debugging (63916) was removed and the servo changed it back to a value that minimizes B1p, as it is supposed to.

We reloaded the filters in ASC_pre to make this solution permanent. The new value of the demodulation phase is saved in ASC_pre but the config file has not been reloaded to avoid the risk of an unlock. Since Metatron does not change it, the value we set online should remain until there is a chance of reloading ASC_pre safely.

It should work as intended now (the servo brings the setpoint to a value that corresponds to the minimum of the "ball" that appear on the istogram with SDB2_B1p_DC and ASC_DIFFp_TX_INPUT, Figs.1,2), but we will keep monitoring it over the next few days.

Today the DaqBox 169 was replaced with the DaqBox 126 with the following mezzanines:

• Slot0: mezzanine ADC +/-10V nbr 84
• Slot1: mezzanine ADC +/-10V nbr 188
• Slot2: mezzanine ADC +/-10V nbr 252
• Slot3: mezzanine ADC +/-10V nbr 262

The DaqBox 169 was not working anymore since yesterday. The DaqBox was showing abnormal functioning since March 21.
https://logbook.virgo-gw.eu/virgo/?r=63851

This DBox destruction is explained by mice (or other animals...) which defecated on the DBox.

The DBox 126 was reconfigured at 9h44m40 UTC.
2024-04-09-09h44m40-UTC    info letendre     'Reload config' sent to SWEB_dbox_rack

An updated version on the Tolm network

The ServerMoni configuration has been updated by increasing the threshold from 20ns  to 30ns for the Tolm devices using the IRIGB

Operations performed at  2024-04-06-05h50m41-UTC

The DaqBox 169 named "WEB_DBOX_ADC1" does not work properly since March 21 18:00. The mezzanine ADC in slot 0 "WEB_ADC_MONI3" does not work. This mezzanine has no +12V power supply anymore.
Some signals of mezzanine ADC in slot 0 "WEB_ADC_MONI0" also show a strange behavior.

Today I deactivated the mezzanine in location 3 because there is a large current consumption on the -12V power rail due to the lack of +12V.
2024-04-05-07h56m45-UTC    info letendre     Reconfigure WEB_DBOX_ADC1 in SWEB_dbox_rack

The problem could be due to some animals dropping on this DBox...

We will change this DBox during the next maintenance shift.

All the rtpc4's Acl servers have been restarted with the latest Acl version v1r24p17 .

Operation complete at 2024-04-04-15h41m01-UTC

The 2D scan has been started after the work of Alain.

the input parameters are :

Amplitude 0.3

duration 10 days

This afternoon, we have changed the data type of the calibrated DC channels starting with prefix "DET_". These channels data are now written as double.

This concerns all the channels named as DET__DC, where can be: B1, B1p, B1s, B5, B2, B4, B7, B8.

To this purpose the photodiodes processes in ACL had to be restarted. Actions were completed between 14h26 and 14h34 utc.

After the maintenance, we have had mainly two types of unlocks: some at CARM_MC_IR, which we didn't pay much attention since the seismic and wind activity are high. However, every time that we rach CARM NULL 1f we would unlock. We could see many "oscillations" on all the longitudinal loops, the PR angular loops and then saturation of B2 photodiode.

It took some time to understand the reason for these unlocks, but since they were all of them at the same moment Diego remembered that it was the moment when we enagge the CARM slow loop. Indeed it looked like it was oscillating at veyr low frequency, so we increased its gain and it worked (1/1 times, we need to see if this works in the long time). Figure 1 shows the clearest example.

WHile debugging I profited to tune the demodulation phase of MICH and SRCL on DRMI 3F, which was off by 0.2 only, then I checked it at STEP 3, and CARM NULL 3f and it looked well tuned. I also fine tuned the DARM phase in STEP3, which ws off by 0.3.

This doesn't explain the unlocks of yesteday on STEP 2 and STEP 3. We keep on investigating.

Remarks related to the DAQ report

•  The issues related to the MdVim are not yet understood: investigations, in-progress
• The FmTrend have not been restarted: it's running since 2024-03-19-18h14m18-UTC

As the MdVim host (olserver117) is unreachable since 04h30-LT, to allow the  update of the VIM page

• the servers previously running on the olserver115(16cpu)  have been moved on the olserver119(4cpu): FmRawBack and MdDQ
• the servers previously running on the olserver117(16cpu)  have been moved on the olserver115(16cpu): MdVim

Operation performed at 08h00-UTC

As usual , the logbook report stayed in the draft instead to be posted