As commented this morning at the Commissioning meeting, this 60 Hz bump (peak at 62-64 Hz, width of ~8 Hz) had already been present since at least April 2024. Then it was probably absent for about two months, and reappeared on June 21st.

Figures 1 and 2 (detail of 1) show it at 5 characteristic times in which it was easier to identify it. The backline is the O4b reference sensitivity shown on VIM. There is no guarantee that this bump wasn't there even before April.

To identify it, I used the SANDI - SpectrAl Noise Difference Identifier daily monitor (which unfortunately only runs from the beginning of O4b).

There is suspicious correlation between this bump and the vibration noise measured by accelerometers in the "PR" area. Figures 1 and 2 are done using Didier's long spectrogram script reading the VIM spectrogram recorded data of Hrec_hoft and the accelerometer ENV_PR_ACC_Z. The result is shown in Figure 1 (data from June 21) and Figure 2 (data from April 12, which is the other time window indicated by Francesco). The appearece of the Bump in Hrec  and PRCL and also its shifting in frequency (62 to 64 Hz) is correlated with an excess vibration noise at that frequency measured on PR vacuum chamber. Note that other nearby accelerometers (SPR_ACC, and IB_CT_ACC for example) also see noise from the same source, however the SPR_ACC measures the larger vibration level (Figure 3). The fact that the bump is seen (but not coherent) also in PRCL was a good hint to search this area.

I tried to investigate the fluctuations of this bump by estimating with BRiSTOL the Band Limited RMS of Hrec in the [59,65] Hz band from 21 June to 2 July. Then, drawing upon the hint by Irene and Paolo's message, I brute force correlated this BLRMS with all the (1454) ENV channels in the trend frame. About forty of them were found to have a Pearson correlation coefficient larger than 0.2. Attached you can find the text file with these channels.

As already observed, the most correlated channel was found to be a (vertical) seismometer in the central building. Many other seismometers are similarly listed in the attached list, including the EDB seismometers, as well as EQB1 and EQB2 ones: Figures 1, 2, 3 and 4. Surprisingly, a quite large correlation, as well as visual similarity, resulted with IPS voltage: Figure 5. I don't know if this correlation is due to the proximity to 50 Hz, which ends up also contaminating the bump band of interest.

Another channel with an interesting correlation (but whose origin I don't know, even if I suspect an electrical one...) is ENV_MCB_ADC_EMPTY_rms: Figure 6.

There is also something around 62 Hz, but more fluctuating in frequency, in ENV_LB_MIC and ENV_LB_ACC_Z (plot1 and plot2).
Not sure if this can be related to what is observed in ENV_PR_ACC_Z and in h(t) (plot3 shows that there is no coherence around 62 Hz
between ENV_PR_ACC_Z and those two channels), but I add anyway this information.

Figure 1. The bump at 63Hz is also visible on B4 DC, and there is a small amount of coherence (5-10%) between B4 DC and h(t) for that bump.

It is unlikely to be the source, but it looks a bit similar to when step motors are enabled on mini-tower suspended benches. But these where causing a much louder bump at ~80Hz, an example from 2017: https://logbook.virgo-gw.eu/virgo/?r=38868

The CEB NN array gives us some hints on the location of the noise source:

Figure 1: it is a CEB NN seismic array map where the colorbar is the ASD of geophones at 62 Hz. We noticed that the geophones around WI  (#209, #214) and PR  (#239) towers show the largest amplitude.

Figure 2: it shows the spectrograms of the geophones in this area and the spectrograms of the accelerometer on the PR flange and the SPR flange (elog 53741). This figure confirms the indication of the noise source given by the map of the seismic array in Figure 1.

The spectral amplitude 2.5e-4 m/s^2/sqrt(Hz)  measured by SPR_ACC at 62 Hz converted in velocity units is 6.4e-7 m/s/sqrt(Hz). It is consistent with the amplitude measured by geophone #209.

We also checked the microphones and magnetometer arrays but observed no similar noise path.

It seems that the 63 Hz bump has a shallow companion at the double frequency that is ~125 Hz, which is also faintly seen in B4.

Figure 1 compares spectra of hrec and B4 at two periods with low microseism, with the primary bump slightly shifted in frequency (as already noted the shift is associated to a corresponding frequency shift of the vibratuion noise). The secondary bump follows.

Figure 2 is similar, but it compares one period with low microseism and one with higher microseism (from 1E-6 to 3E-7 m/s brms 0.1-1 Hz). The primary bump becomes wider, the secondary *seems* too.

I could not find in seismic sensors (PR area) an harmonic of the primary vibration noise, although it might exist. Alternatively, the second bump can be explained as a non linear effect from the primary vibration, which is typical of scattered light noise (such behaviour has been observed in the past). However, reading from the vibration sensors in the area where the vibration is most intense (NN_CEB_ACC_Z_209 on ground, ENV_SPR_ACC_Z on vacuum chamber), the rms level of the noise is of the order of 1E-8 m, which would not justify nonlinearities, unless the vibration noise is amplified at the actual scatterer, for example if it matches a resonat mode of a mount or else. 

The goal of the shift was to look for the noise source of the 63 Hz bump and the coupling location. We performed the following actions.

• We installed a beam dump at the input of EPRB, Fig 1. After the relock of ITF, the bump on Hrec was no longer visible, Fig 6 and Fig.7 (compare purple vs orange). It is worth noticing at that time, the noise level of the 63 Hz vibration was smaller on all accelerometers. To validate the effect of the beam dump, we placed the shaker on EPRB and generated different kinds of vibration:
  • sweep in the free. range (60-65) Hz
  • lines at 60Hz and 62 Hz
• Although the vibration level on the bench increased by at least 10 times, we did not generate any bump in Hrec, blue curves in Fig.6,  blue and green curves in Fig.7.
   
• We reconnected the vertical accelerometer ENV_EPRB_ACC_Y which was already on the bench but not acquired, Fig 1. Comparing the ASD, we found that this accelerometer measures the 63 Hz vibration 10 times larger than the SPR accelerometer.
   
• We used the accelerometer of the INJ cryotrap ENV_IB_CT_ACC_X as a sensor to perform our sniffing investigation.
   
• We found the NN geophone #209 in contact with the metal H-beam. We moved it to the floor nearby and in this position, the geophone did not see any more vibration at 63 Hz,  Fig 2. We deduced that the vibration is transmitted through the H-beam.
   
• We disconnected the cables of the piezo of EPRB without effect on EPRB/SPR accelerometers. We can exclude this source of vibration. The cables have been reconnected.
   
• We tapped the EPRB and we observed the excitation of the bench resonances at 60 Hz and 110 Hz, Fig 3. This explains why the 63 Hz vibration is amplified on the bench. 
   
• For the sniffing, we installed the ENV_IB_CT_ACC_X on the base of the WI SAT rack and on the Sa suspension crate, Fig. 4. To understand if the noise is associated with the SAT cooling fans,  we asked Alberto G. to increase the speed. No effect is observed on 63 Hz vibration on EPRB, therefore we can exclude this noise source, Fig. 5.
   
• As a last action, we moved the small shaker on the PR tower base. The injected line at 60 Hz is well visible in Hrec (Fig. 8) as a narrow line. This seems indicating a different coupling path (linear), which is not the non-linear path due to EPRB scattering. It is worth further investigation.

We concluded our investigations around 19:30 LT because the ITF was repetitively unlocked, not because of our actions.

At the end of the shift, we removed the beam dump and shaker from EPRB. It would be interesting to repeat the bench shaking without the beam dump.

Today, during maintenance, we continued the investigation for the 63 Hz source.

We again used an accelerometer, borrowing the ENV_IB_CT_ACC_X channel (temporarily renamed to ENV_SNIFFING_ACC).

We took a look around the PR, BS and WI towers, measuring many (~20) locations/devices at ground and platform level, but without success: no obvious source was found.

We left the temporary accelerometer on the base of the BS rack, near the RH power supplies. We plan to remove it (setting back the accelerometer names) at the first opportunity.

p.s. the reduction of noise visible in NN_CEB_ACC_Z_214 around 09:30 UTC is due to the fact that it was touching the metal structure with its cables; we rearranged them and now it only sense the vibrations of the concrete (as it is expected to do).

just adding a few information: we found the noise is sensed the most intense on the platform around the SPRB minitower, yet not on the SPRB chamber. The noise is also sensed on the H-bar structures around the PR and WI towers base. It seems we can exclude the TCS area. We sensed several sources, which we exclude: cooling fans of the link turbo pumps, the link turbo pumps, it seems we can also exclude the TCS area. We inspected carefully the area among the towers at floor level and the few powered devices that we found have no moving parts. Federico also inspected the area around the SPRB minitower (the front part, under the tent) which is empty.

Cooling fans of the RH power supply module produce a main vibration peak at 62.4 Hz, which is indeed among the noises sensed by the EPRB accelerometer (see the Figure attached in the previous elog) but it is not the one we are looking for, which instead moves in time in a peculiar way (see the same Figure).

We find useful to look at the signal for a longer time. It would be useful, when possible, for example if an unlock occurs, to move the accelerometer probe on the nearby BS SAT rack, which we tested but with unclear result. 

Figure 1 shows the evolution of the 63 Hz vibration noise over the last two weeks. We notice that the vibration attenuated on Tue July 9th and reappeared on Tue July 16th.

These days coincide with our hunting activity. We tracked these occurences at 7:32 UTC on July 9th (Figure 2) and at 7:50 UTC on July 16th (Figure 3 and Figure 4(zoom)). The black arrows point the time when we see the 63 Hz noise descrease (or increase). Note that this identification is not easy since the source moves a lot in frequency.

Both events associate with a large transient vibration sensed by the accelerometers in the SPR area. At both times there was activity inside the CEB hall (cryotraps refill and maintenance). One action that is close in time with these event is Federico disconnecting the cable of accelerometer ENV_IB_CT_ACC from the power supply box (blue PCB box) located on the base of IB tower (facing EEroom). This time is well identifiable in spectrograms because right after the IB_CT acc signal goes off. It is a bit strange that this action could have been the cause. We have to investigate further, tray to repeat the action. One interesting observation is that this change of intensity did not occur in the previous maintenances, but it seems linked to our presence.

We also notice (Figure 2) that the signal intensity reduced in accelerometers attached to vacuum chambers but not in the NN sensor placed on the floor. This makes us think that it was not the source that changed intensity but more likely a change of the propagation path or a shortcut removed/added.

Evolution of the bump in the last two weeks

The plot (Fig.1) shows the evolution of the vibration noise at 63 Hz (the accelerometer used is attached to the SPR chamber right above the EPRB) and the corresponding evolution of the bump structure in DET_B4. The correlation is evident.

The bump in Hrec seems there as well but much shallower, thus difficult to be visually spotted. Figure 2 compares DET_B4 and Hrec ffts at one given time.

To note, that a line at about 62.4 Hz is present in Hrec from time to time and it is more evident when the bump is not present (because of the 63 Hz vibration noise being less intense). This line has small coherence with accelerometer sensors and other ENV sensors, also spotted by the Bruco: https://scientists.virgo-gw.eu/DataAnalysis/bruco/2024/2024-07-11//Hrec_hoft_1404699359/bruco.html.  Might it be related to the coherent noise that we noticed when injecting a sinusoidal vibration on PR base? To be investigated...  

note: the x.axis label of plots in Figure 1 is wrong. Should be "time (2000s chuncks)", it goes from July 7 00:00 to about July 17 at 12:00 

"We find useful to look at the signal for a longer time. It would be useful, when possible, for example if an unlock occurs, to move the accelerometer probe on the nearby BS SAT rack, which we tested but with unclear result."

Done this morning, between 09:25 and 09:30 UTC

In the new position the 63 Hz bump (today at 62 Hz) is visible, but with an amplitude that does not seems compatible with the source location (the color scale is the same)

A comparison between B4 and Hrec. After July 16 the bump reappeared in B4 with similar characteristics, while in Hrec it is less much less evident.

Figure 2 compares Hrec, B4 and SPR accelerometer spectra at a time on June 29 and a recent time of July 18. The two times have been chosen such that the amplitude of the B4 bump is similar. 1hour average is used for both. The amplitude of the vibration noise, as measured by the SPR accelerometer, is similar for both times. The amplitude of the bump in Hrec is lower. The indication from this plot is that the coupling to B4 has not changed while the coupling to Hrec has reduced. We do not know what might have caused this reduction.

In Hrec, note the line at 62.4 Hz which seems to have a slow amplitude modulation (Figure 1).

Taking the profit of these upgrading days, we further investigated the noise source of the 63 Hz bump. From the offline analysis, we noticed:

• the geophone #212 (positioned on the West side of the WI tower) and #239 (positioned between PR and BS tower) measured this noise, Figure 1. The noise also has an acoustic component visible in microphones  #207 (level 2, located between INJ and PR towers)  and  #213,214, 223 (level 3 - scaffolding platform, BS, WI, INJ towers respectively), Figure 2 and Figure 3.
• Looking at the spectrogram of ENV* microphones and accelerometers, we observed this noise is also polluting other sensors: the mic of EIB, accelerometer on TCS WI bench and accelerometer of INJ cryotrap (*IB_CT_FINGER*), Figure 4.

Following the results above, we decided to sniff the scaffolding platform, in particular the SAT racks of PR and WI towers which are in the incriminated area. As reported in the attached file, we placed the accelerometer (ENV_SNIFFING_ACC) in different positions on the racks, Figure 5.
The culprits are the cooling fans of the PR SAT crates. We also observed the following:

• the noise measured on the PR SAT crates corresponds to the noise seen by EPRB, Figure 6
• we could not disentangle between the fans of the top crate (Sa) and bottom crate (Sc); this could achieved by changing the speed of the single fans modules (by SAT)
• it is worth noticing the fans are working properly, nothing is breaking. We tested similarly also the WI SAT crates and we found a similar behavior (frequency oscillation), right box of Figure 7.  It just happens that the frequency of PR cooling fans matches the mechanical mode of the EPRB bench (60 Hz)
• Akis confirmed that all fans' speeds are the same and they run in feedback to keep the crate temperature stable.
• the fans are assembled in 4 planes, 3 fans each, 2 planes per crate

This morning, we installed two accelerometers near the cooling fan of the power supply of SAT Sc_PR (bottom) and Sa_PR (top) crates, see attached Figures. The channel names are ENV_PR_RACK_Sa_ACC and ENV_PR_RACK_Sc_ACC.
The preliminary results are not conclusive. The wondering 63 Hz peak is seen in both accelerometers with an amplitude similar to the sniffing accelerometer.
The higher frequency components of the 63 Hz wondering line are a bit more intense in the Sa_PR, see Figures.
 

Profiting of an unlock during the ENV shift at about 17:20 UTC we added one microphone right in front of the cooling fan of the power supply module of the Sa crate of the PR SAT rack. The name of the channel is ENV_PR_RACK_Sa_MIC and replaced channel ENV_PR_RACK_Sc_ACC.

We followed the prescriptions described in the git task.

At first we observed that during quiet time the 63Hz bump was not present both in Hrec and B4_DC signals.  A quick look at VIM spectrograms showed that the bump was present at least in B4_DC signal up to before yesterday maintenance. This has to be further investigated.

We proceeded with injecting vibration noise with the shaker (which is positioned on EPRB since yesterday). As a first attempt we injected a line at 63 Hz (Figure 1). However the volume was set a bit too high and we lost the B4 beam signal after 13s of shaking, the ITF unlocked right away.

Once the ITF relocked we injected a colored noise shaped between 50 Hz and 70 Hz at what we judged to be the maximum volume allowed. This injection lasted 300s. This injection produced a corresponding bump in B4_DC but not in hrec (purple lines in Figures 5 and 6).

Camilla added an high absorption optical density filter (transmission 1/1000) in front of the razor stack beam dump which blocks the beam from the scanning Fabry Perot, and closed a bit two irises along the path of the phase camera beam. She also had to move the shaker in another positon on EPRB because of space constraints. Figure 2 and Figure 3 are picture taken respectively before and after these actions. The ITF did not unlock.

We repeated the injection of the same colored noise (same shape and same volume). The bump excited on B4 changed slightly shape and size, but this seems to follow a similar slight reduction in the excitation sensed by the EPRB accelerometer (orange curves in Figures 5 and 6).

Camilla performed the second part of the planned intervention. She added one optical density with 30% transmission of the phase camera beam path. Figure 4 is a picture taken after this actions.

Again we repeated the injection of the same colored noise. We did not observe any reduction of the bump we excited in B4: see the blue curves in Figures 5 and 6.

We moved the shaker on the PR tower base (West side) and repeated the injection of a few lines around 60 Hz in the attempt to investigate further what obseved in  https://logbook.virgo-gw.eu/virgo/?r=64685

Precisely we injected lines at 70Hz, 60Hz, 65 Hz, and 64 Hz. When injecting at 60 Hz we see clearly a line in Hrec (and also in B4): see Figure 7. Lines at 70, 65 and 64 produced someting in B4 Hz. Finally the injection of the line at 64 unlocked the ITF.

Since no clear benefits after the installation of the density along the phase camera beam path (OD 0.5) had been observed this morning, while the ITF was in down due to the earthquake, I removed it (12:15 LT).

We installed a second microphone in front of the cooling fan of the power supply module of the Sc crate of the PR SAT rack.  The name of the channel is ENV_PR_RACK_Sc_MIC and replaced channel ENV_PR_RACK_Sa_ACC.

Yesterday between 17:56 and 18:07 we performed a few tappings in the area between PR and BS, East side. With the little hammer (impact hammer PCB mod.086C01 ) we did a sequence of gentle hits on the following:

• PR blide flange: Figure 6
• PR-BS link pipe (middle): Figure 7
• SPRB East flange: Figure 8
• PR-SPRB link pipe: Figure 9
• PR-BS link pipe, near bellow: Figure 10

We could see some noise excess in B4_DC, but nothing in Hrec. In B4 we see rising one existing structure at 144 Hz, and some new ones appearing at 110 Hz (small) 175 Hz  and 287 Hz.

• tapping on Blind flange: Figure 1
• tapping on PR-BS link pipe: Figure 2
• tapping SPRB East flange: Filgure 3
• tapping PR-SPRB link pipe: Figure 4
• tapping PR-BS link pipe near bellow: Figure 5
• spectrogram including the 5 tappings: Figure 11

Conclusions:

• The impression is that among these position the slightly more senitive one is the PR blind flange.
• We could not continue exploring other locations on the West side because the ITF unlocked.

The change in the vibration noise transmission thorought the PR - SPRB area happened again during the last Tuesday maintenance (Aug. 6th) as shown in the attached specgrograms.

In the first spectrogram, the top channel is the accelerometer monitoring the 63Hz vibration source at the PR SAT rack. The intensity of the source did not change. Yet, after 9 AM UTC the vibration noise transmitted to the EPRB bench reduced significantly. It is difficult to identify when exactly this happened between 6AM and 9AM. That morning Maria and myself were on the platform near the PR crate installing sensors. We might have changed the shortcut for the PR crate vibrations by simply walking around there. We also sat on the platform steps on top of SPRB, but we did not touch any cable. The area is tightly packed and the inspection is not easy, yet we have the impression that the shorcut should be around there. Other people were around, for example for the CT refill, we cannot exclude it might come from their actions.

Figure 1 also shows that consequently the noise bump in B4 which was there before the maintenance it disappeared afterwards.

The second spectrogram shows that the 63 Hz vibration noise actually reduced not only at EPRB but aslo at several sensors in the PR-SPRB area.

on July 31 a test was done  changing the speed of cooling fans of PR SAT rack. The speed of the fan was changed from 2200rpm (37Hz)  2600rpm (43Hz) and then back. The action was done separately on the two crates. First the fans of the Sc crate, from 10:40 to 12:30 UTC, and then the fans of the Sa crate from 12:50 to 14:00 UTC.  

Figure 1 shows this action seen by the accelerometer positioned on the rack.

We observe that a set of lines moves from approx 36 Hz to approx 43 Hz at both times. We interpret that these are the vibration noise associated to cooling fans. Correspondly we observe the "63Hz" noise on the reference EPRB accelerometer. Its frequency, apart of the usual fluctuation, did not change. We interpret that this vibration noise is not due to these fans. 

Yet we observe that the test produced some influence on the fluctuation pattern of the 63 Hz. Observing Figure 1, this pattern change is more evident during the test with the Sa crate.

Additionally, looking closely in Figures 2 and 3, we can observe that while most of the cooling fans lines are quite stable, there is one "anomalous" line in the bunch which has a fluctuation pattern, and this pattern is a mirror image of the pattern of the 63 Hz noise. This anomalous line is associated to the Sa crate.

After inspection Federico found another fan located on each crate: the fan that cools the power supply (PS) module.

One possible interpretation of the data is that the anomalous line is associated to one cooling fan of the Sa crate which is influenced (in counter action) by its  PS module fan: when the PS fan goes faster fluxing more air the anomalous fan is affected and fluxes less. This would lead to the conclusion that the source of the 63 Hz vibration noise is the cooling fan of the PS module of the Sa crate. But, there can be alternative interpretations.

Taking advantage of the maintenance, we carried out a visual inspection around SPRB-EPRB. We found a small ladder near the SPRB tower. The ladder is anchored to the H-beam and touches the SPRB flange. We think this may be a short-circuit point. Further investigation will follow.

We reconnected the accelerometer ENV_IB_CT_ACC. We left the sniffing accelerometer on the PR rack for future investigation. It is not acquired.

The goal of the shift was to eliminate the SPRB flange shortcut by removing the ladder. Before doing so, we wanted to insert a piece of Sorbothane between the ladder and the flange, but we realized there wasn’t enough space. At the same time, removing the ladder seemed a bit risky due to the limited space and cables around it. We believe the action requires an area inspection before removing the ladder.

Today at 10:45 UTC we again disconnected (and “commented out”) ENV_IB_CT_ACC, using its harware and ADC channel to monitor again the ENV_SNIFFING_ACC accelerometer that had been left on the PR rack.

The goal is to have a monitor while performing some work on the ladder that presumably creates the shortcut.

Today we made the test of "forcing" the ladder in a new position that would not touch the SPRB flange.

• First we placed a small bluetooth shaker into PR rack, where we have our witness accelerometer (ENV_SNIFFING_ACC).
• Then Maria went near the ladder trying to move it.

The test is quite inconclusive: we did not had any 63 Hz bump variations in the accelerometers (ENV_EPRB_ACC_Y and ENV_SPR_ACC_Z) during the test (red square in the attached figure).
At the same time we noticed that there are bunches of cables that create a shortcut between the "ceiling" and SPRB vacuum chamber. We think these are also a shortcut.