Reports 1-1 of 1 Clear search Modify search
Virgo Runs (ER16)
berni - 23:00 Wednesday 03 April 2024 (63831) Print this report
Operator Report - Afternoon shift

ITF found in commissioning to allow Valerio to work at the PR suspension; activity concluded at 16:18 UTC.

After that the ITF was relocked without problems and INJ team started to work at INJ noise hunting shift - RFC alignment and SBE EIB noise injections in LN3.

In parallel Franco and Diego worked on the automation.

All the activities concluded at 19:16 UTC so I set calibration and I performed the DAILY_CALIBRATION, completed at 19:30 UTC.

ITF in Science mode at the 19:32 UTC; still in Science at the end of the shift.



The red flag Timing_dsp Daq_Sa_PR_Tolm_Send..Start_Min that appeared after the intervention on the PR suspension; experts have been informed.

Images attached to this report
Comments to this report:
narnaud - 23:33 Wednesday 03 April 2024 (63833) Print this report

For the record, there have been problems with h(t) between ~1230 and ~1730 UTC, likely a fallout of the automation work, as pointed out at the end of entry #63828. In particular, low-latency h(t) frames contained the state vector but not the h(t) channel itself. The problem was reported by PyCBC Live around 1650 UTC and manifested itself as a much larger latency for the pipeline. They posted their observation on Mattermost and opened an issue on Gitlab. The Virgo control room was informed about 10 minutes later by phone and the problem was fixed half an hour later. There will be more investigations to understand what caused the problem and why it was not discovered earlier.

Images attached to this comment
verkindt - 23:58 Wednesday 03 April 2024 (63834) Print this report

Around 12h30 UTC, I have started a new process HrecN, in parallel to Hrec, with a configuration testing the new actuators models provided by the most recent calibration.
But I forgot to change the FDOUT_SHM (the output shared memory where are written the output data). As a result, HrecN was writing its output frames in the same shared memory as Hrec and sometimes, the HrecN output frames overwrote the Hrec output frames.
HrecN produces a DQ_ANALYSIS_STATE_VECTOR channel like Hrec but produces a h(t) channel named V1:HrecN_hoft_16384Hz instead of V1:Hrec_hoft_16384Hz.
As a result, sometimes, the channel V1:Hrec_hoft_16384Hz was not found in the frames received by online data analysis pipelines.
Sorry for the inconvenience. It was anyway a good test showing that sending to online analysis data frames that do not contain the channel V1:Hrec_hoft_16384Hz still implies perturbations.
HrecN configuration has been fixed.
Hrec and HrecN are now running in parallel without interfering. HrecN will be stopped in a few days, once new actuators models have been tested/adjusted over this time.
In the future, any stop of Hrec process will imply that no frame is received by V1FromOnlineR from Hrec, thus frames output by V1FromOnlineR and received by online analysis will be without the channel Hrec_hoft_16384Hz. So, a trivial rule is: Hrec should not be stopped while we are in science mode ;-)

Search Help