Figure 1 shows a steady state of glitches during the weekend, and Figure 2 shows last night that the situation became worse for a few hours in the middle of the lock.
It would be interesting to try to correlate these glitches which could give a lead to the origin of the problem. For example find correlation with glitches in other channels, but especially find if there is a correlation between glitch occurrence and other signals in time domain. For example that glitches occur when a particular alignment signal is large.
The daily automated UPV runs (on the past day and on the past week) based on the Online Omicron triggers don't find anything this time -- they are available on VIM: https://vim-online.virgo-gw.eu/?config=64.
The investigation into this excess of glitches has not found the source yet but is still ongoing. For now, I report the results of a BruCo run around 21:20 on November 26, when the glitch rate was larger. Unfortunately, this test did not result in any witness channels with evident coherence with Hrec.
I couldn't identify any periodicity in the time of occurrence of these glitches. Logging at Figure 1, these look indeed like clusters of energy excesses. So, not finding periodicity in the omicron triggers could be due to the ambiguity in identifying their "central time". I'll further investigate this tomorrow.
A correlation analysis using as a target the RMS of Hrec in the band [20,60] Hz, with lines removed, and all the *_rms channels from the trend frame produced no clear evidence of correlation either. I will repeat this analysis tomorrow with more care. Some "potential witnesses" of these glitches seem to be the accelerometers of the filter 0 of the Beam Splitter suspension. I report here some channels and their correlation but as I said I will redo the test by modifying the target so that it best reflects the energy of the glitches, and possibly the auxiliary channels to correlate.
Channel name Pearson coefficient
V1:Sa_BS_F0_KACC_V2_rms: 0.394
V1:Sa_BS_F0_ACC_V2_rms: 0.394
V1:Sc_NE_MIR_Y_AA_rms: 0.393
V1:Sa_BS_F0_KACC_V1_rms: 0.393
V1:Sa_BS_F0_ACC_V1_rms: 0.393
V1:Sa_OB_F0_Z_GIPC_rms: 0.370
V1:Sa_OB_F0_ACC_X_rms: 0.355
V1:Sa_BS_F0_ACC_V1_FB_rms: 0.351
V1:Sa_BS_F0_ACC_V2_FB_rms: 0.350
The horizontal and vertical accelerometers of BS and PR have a lot of signal correlated to the locking signals, since we started to allocate the locking to IP. Maybe this can induce some correlation between ITF glitches and F0 ACC signals, without being the cause of the glitches. It also true that F0 ACC signal is much much larger when the locking is allocated to IP: this is a new working condition which could determine some unexpected drawback, not easy to be identified in the local signals. An eventual malfunctioning would be not systematic, because the strategy has been used with 55 Mpc
In phase of investigtion, we can turn off the allocation to PR and BS IP and see what happens.
The glitches in excess seem to be mostly gone: see omicron-plot in attachment.
Continuing the investigations into this new family of glitches, I first tried to figure out which channel best represented them. The most straightforward choice was some BRMS of Hrec in the 20 to 80 Hz region of the bandwidth of these glitches (refer to the spectrograms in the previous comment), where no spectral lines were present. Among these, V1:DQ_BRMSMonHrec_BRMS_HREC_HOFT_FREQ_BAND_45_47_Hrec_hoft_16384Hz , from the trend frame, was selected because the 45-47 Hz band is sufficiently wide and almost free of lines. Before attempting correlation studies, beware that these BRMS channels have a 10-second delay, due to the window used to estimate them; despite the 1 Hz sample rate, they appear as step functions that update their values every 10 sec. This rate seems too large to investigate individual clusters of excess of energy (duration ~2 sec, separated by ~10-30 seconds, as in Fig. 1 of the previous comment) but can give an average estimate of the excess of noise. To have a better resolution, I also investigated the following possibilities:
- Whitened Hrec data (to remove the stationary part of the noise, including spectral lines) bandpassed between 20 and 80 Hz, from which I obtained the RMS on strides of 1 or 2 seconds
- A BRMS obtained by summing the spectrogram of the whitened data (Q-scan) over the frequencies. This method should be approximately equivalent to the previous one (by Parseval's theorem) except for the difference in the spectral estimate.
The results, compared with the Q-scan of a series of these glitches are reported in Figure 1. The similarity between the two previous methods (orange and green curves) confirms the expectation.
To investigate potential periodicities, I estimated the ASD of these series: Figure 2. None of them shows any peak or structure in the spectrum, hence no periodicity. In the series of the BRMS ASD, the lobes represent the actual sampling of 0.1 Hz (Nyquist frequency at 0.05) as commented above.
I then tried to use both the BRMS time series, corrected for the 10-second delay, and the one obtained from whitened data to look for correlations with all the *_rms channels in the trend frame. For the first one the results are almost zero, while for the second one, the usual correlations with the PR and BS accelerometers were found. I have attached the list for completeness.
Things remained to test:
- A more sophisticated correlation between any of the above series and some transformation of the auxiliary channels from the raw frame
- Correlation of the glitch rate time series with some other slow monitor, to find what could be the cause of the increased rate, if not of the glitches themselves
The rate of these mysterious glitches is correlated with the noise at West End.
I took the omicron triggers of Hrec starting from November 23rd. I filtered them for SNR > 6.5, to avoid the "noise tails", and for center frequency less than 100 Hz. From them, I obtained the time series of the rate per minute. I then did a brute force correlation with all the rms and mean channels (14k) of the trend frame. The results for Pearson correlation coefficient greater than 0.2 are reported in the attached file. I also report some plots of the most correlated channels.
Figure 1 compare the SNEB and SWEB angular control in the past 24h. The SWEB peak-to-peak angular motion and correction are at least 10 times higher than for SNEB.
Figure 2 compare SNEB and SWEB in frequency domain. The local control sensing noise for SWEB is over an order of magnitude higher than for SNEB at high frequency (above 5Hz).
A similar problem has occurred a year ago, and was investigated for two weeks. The series of logbook entries is not completely clear but it seems the problem was resolved by changing the elapsed time in some of the computations of the Pcal https://logbook.virgo-gw.eu/virgo/?r=62484
Figure 3 The same broad peak at ~10kHz is present on SWEB outer coils as a year ago, close to the narrow line used to demodulate the local control error signal.
Just to be sure and for the record, I looked into the PCal glitches at WE if there were any coincident with those recent glitches in Hrec.
Figure 1 shows the Hrec scalogram and figure 2 show the one for PCAL_WE_Rx_PD1_DC and the glitches do not seem to coincide either in frequency or in time.
The attached plot is a cleaned BRMS of Hrec in the band 45-47 Hz, observed in the latest 17 days. Today, for the first time, the level of noise in that band seems the same as 17 days ago.
That's excellent news! I confirm that the glitches seem to have disappeared after today's maintenance and in particular once we returned to Science mode at 17:48 UTC. This seems to be the case for the frequent, low-frequency glitches, identified as being somehow related to SWEB controls (#65654), but also for the rarer, high-frequency ones (#65646). Of course, the 25-minute glitches, visible as a continuous trigger at 40-50 Hz with SNR ~400, are still there.
Figure: glitchgram with the omicron triggers from the beginning of December to now, and below the rate series. It is evident that from a rate even larger than 10 per minute we have returned, in the last hours, to a value comparable to the median value of O4b of 0.1/minute.
The SWEB control issue has been resolved on Friday, and the glitches stopped on Tuesday evening. So the correlation between glitches and SWEB control issue is likely to be a coincidence. It would be good to check using data since Friday last week that indeed the correlation was just a coincidence and is not supported by data since Friday.
...and the 20-80 Hz glitch issue seems to have reappeared in the lock of this morning between 9:35 and 10:25 UTC. I'm resuming the investigation.
The first hour back in Science seems quiet: https://vim-online.virgo-gw.eu/resources/2024-12-04/resources/20241204_ana_plot_Hrec_hoft_16384Hz_freqtime.png.
