At the moment the tiltmeter is still unlocked. We will expect to lock the tiltmeter tomorrow.
The first unlock occurring at 14:41:11 UTC, July 15th, is apparently due to a glitch in the INJ system, which triggers an unlock of the SSFS, even though the power stabilization stays locked: see Figs. 1, 2. A few seconds later, at 14:41:14, a series of glitches pollutes the PSTAB_PD2_AC_MONIT signal, see Fig. 3.
The second unlock, occurring at 16:07:34 UTC, is more resemblant to the majority of fast unlocks, throwing also the PSTAB loop out of stability, see Figs. 4, 5.
During the lock stretch lasting about 1 hour from 13:10 UTC today, July 15th, two deep sensitivity drops occurred.
Neither of them seems correlated with PSTAB issues (Fig. 1) or with seismic noise (Fig. 2).
Movies of hRec and DARM spectra, in Mov 1 and Mov 2, show that lower frequencies get suddenly excited, for reasons unclear to me.
Both these unlocks are correlated with glitches in the INJ system
- 11:27:40, Figs 1, 2; no sign of the impending unlock in INJ data before the glitch.
- 13:55:16, Figs 3, 4; ditto, no warning sign.
They are very similar to the fast ones listed in entry 46370.
When Friday Fiodor tried to inject squeezed light into the ITF with the automation the system fall in stuck.
The problem was that the SQZ_MAIN node in the State SQZ_INJECTING (index 120), thus the ITF_LOCK node was in stuck in the state ENGAGING_SQZ (index 167).
In the state SQZ_INJECTING, SQZ_MAIN checks if the CC is closed, the AA is closed and the squeezed vacuum states are correctly injected into the ITF:
- from line 782 to 786 the CC is checked
- from line 788 to 795 the AA is checked
- from line 803 to 808 the level of the 7MHz mag is checked (this is a double check on the AA and on the CC)
Friday I forgot to move these controls from B1_PD1 to B1_PD2. The same adjustment must be performed in the state SQZ_INJECTED (ndex = 130)
Here below the list of modified lines:
- line 782 and 840: from sqz_b1_pd1_I = ezca.get_channel('SQZ_B1_PD1_7MHz_I_50kHz') -> sqz_b1_pd1_I = ezca.get_channel('SQZ_B1_PD2_7MHz_I_50kHz')
- line 803 and 862: from sqz_mag = ezca.get_channel('SDB2_B1_PD1_7MHz_mag_FS_50Hz') -> sqz_mag = ezca.get_channel('SDB2_B1_PD2_7MHz_mag_FS_50Hz')
I'm preparing an update of the SQZ_MAIN node that performs all the adjustments descirbed int this entry and in the Fiodor's one only by changing the parameter "PD_select" in the SQZ_MAIN.ini file (line 63 section [SHUTTER_OPEN]).
This morning I' ve worked on the lock acquisition since the ITF was not locking.
I had to increase the low noise DARM gain and in LOCKED_OMC1_B1s2_DC(GuardState) from 3.5 to 4.2 (more than usual ugf ~110Hz) to be more robust.
Moreover I have tuned the BS setpoints.
The last thing that I've changed, never tryed, is to engage the BS in full bandwidth from LN1 to the OMCs lock, then put back in drift control during the marionette reallocation and in full bandwith @ LN3.
the lines changed are:
line 2763... of ITF_LOCK.py
#### TEMP put 1 if is unlocking
BS.ramp_attribute("AATX_FLAG",2,0)
BS.ramp_attribute("AATY_FLAG",2,0)
if it is not locking put 1.0
line 3406
BS.ramp_attribute("AATX_FLAG",1,0)
BS.ramp_attribute("AATY_FLAG",1,0)
(it can be left)
ITF found in Science Mode, unlocked at 5:04 UTC due to a Fast Unlock. Back in Science Mode at 5:37 UTC.
ITF unlocked again at 6:23 UTC for another Fast Unlock ( see entry #46370 for the unlocks investigation ), after that the ISC team started working on the locking acquisition and BeamSplitter alignment: switched to Commissioning Mode from 7:10 UTC to 10:31 UTC. Back in Science Mode from 10:31 UTC.
Unlocked at 11:27 UTC due to a fast unlock ( see attached plot ), relock in progress.
DET
B5_PD1_Vbias went off at 12:20 UTC. Rearmed and opened at 12:43 UTC.
ISYS
The PSTAB has been opened/closed in order to remove the extra noise. Action performed at 8:07 UTC, 11:38 / 11:39 UTC and 12:31 UTC.
SUSP
WE LC opened at 7:44 UTC after an unlock. Loop closed at 7:50 UTC;
WI LC opened at 8:18 UTC after an unlock, closed at 8:21 UTC. The event happened again at 12:16 UTC, closed at 12:26 UTC.
ISC
(12-07-2019 12:00 - 15-07-2019 10:30) contact the ISC on-call to align the BS
An attempt to deduce the ground angular motion from tiltmeter signals at low frequency has been performed. In order to do that, the mechanical tf, the loop correction and the calibration factor have been taken in account. The result has been compared to a combination of top stage susp sensors, which should be equivalent to an accelerometer placed on the basement, in acc/g units (fig 1). The data are taken in low wind condition, so that the susp signal below 0.1 Hz is quite low (likely limited by the sensor noise). We can see that the tiltmeter sensitivity is still not comparable.
Above 0.2 Hz, where the longitudinal ground motion become dominant on the susp sensor, the tiltmeter shows a quite low signal. At the microseismic peak there is some coherence (fig 2), but the coupling is about 2e-3 (fig 3).
Taking in account data in high wind condition, the expected tilt at 0.1 Hz start to be slightly above the tiltmeter sensitivity (fig 4). In fact the coherence around 0.1 Hz is not zero (fig 5), and the transfer function is about 1 (fig 6).
All three unlocks occurring this early morning, July 15th, are correlated with similar glitches in the INJ system.
They are not preceded by any evident increase of noise levels: very fast unlocks indeed.
On Friday (July 12), to improve the stability of the lock acquistiion during OMC1 and OMC2 locks, we have done additional changes:
1) The transition between DARM SET/GAIN to B1 DC SET/GAIN is now done in a smooth way. This was done by adding B1s2_DC as one of the input choice to LSC_DARM_INPUT, and use it for the initial transfer from the RF error signal to the DC error signal. And reserve B1_DC_INPUT as the signal that has the offset from B1_DC_SET and the automatic gain adjustment. Thir resolve the issue with unlocks in ~50% of cases of the transition that was done instatenously before.
2) We increased the dark fringe offset from 0.225mW to 0.25mW on B1s2. In principle this is to much, as it corresponds to 3mW on the B1 photodiode and the fast shutter on SDB1 closes for 3mW on B1. But because 30% of the light on B1s2 is actually due to HOM and RF sidebands and not carrier TEM00. It actually correspond to ~2.1mW on B1 PD2 DC. This is also made possible as now the transition to remove B1s2 from the DARM error signal is done with a smooth ramp that decrease the dark fringe offset at the same time
3) The state of OMC1 locked with 0.25mW dark fringe offset is not very stable. At 0.225mW we have seen several times that gain of the DARM loop drops (and oscillates at 25Hz), but looking at the RF signal, it corresponded in the one case we looked at a zero crossing of the RF signal, so probably the B1s2 signal is to poluted by junk light and has a fluctuating real offset in meters for DARM
However, I wouldn't remove all the changes once we go back to 2 B1 photodiodes, the new lock acquisition is much more smooth. But it does need cleanup.
I used the recent PSTAB noise activity to test (and validate) the current PSTAB monitoring (see 46334)
The first plot shows the omicron glitch distribution in h(t). The glitches around 20:00 UTC at f~100-200 Hz are caused by PSTAB. Some of these glitches are actually missed by omicron because they are too loud. Looking at the red/green strip chart at the top of the plot, there are many time segments when Omicron saturates (red segments). This is probably caused by loud glitches in PSTAB.
At 22:00 there is also a long segment missed by Omicron. This is also caused by PSTAB malfunctions.
PSTAB glitch activity is monitored by 2 channels:
- V1:OMICRON_RATE_SNR_70_ADC_PSTAB_PD1_AC: measures the glitch rate in PSTAB_PD1_AC with SNR>70
- V1:DQ_Omicron_VETO_GENERIC_PSTAB_PD1_AC_2019-07-05-pstab: flags glitches in PSTAB for which there is a high probability that they couple to h(t)
Looking at these 2 channels (plot 2), it is clear that PSTAB is causing the glitches in h(t). Moreover, the glitch rate measured with V1:OMICRON_RATE_SNR_70_ADC_PSTAB_PD1_AC tells you how bad the situation is.
During the night shift the ITF unlocked once at 3:48 UTC due to a fast unlock (pitcure attached). After that the Science Mode has been recovered at the second attempt (4:26 UTC). Hrec Range BNS ~ 43Mpc.
PSTAB
The PSTAB has been opened/closed in order to remove the extra noise. Action performed at 4:12 UTC.
Guard Tour
22:22 - 22:55 UTC
00:25 - 00:55 UTC
02:30 - 03:00 UTC
04:11 - 04:41 UTC
ISC
(12-07-2019 12:00 - ) contact the ISC on-call to align the BS
The sensitivity drop discussed in entry 46355 was clearly correlated with the loss of PSTAB lock, which motivates looking back to previous days and checking if other, less severe drops are also caused by PSTAB issues.
On July 10th, Fig. 1 clearly shows that sensitivity drops occurring from 16 to 17 UTC correlate well with increased PSTAB signals.
On July 11th, Fig. 2 shows a single, isolated sensitivity drop, unrelated with PSTAB.
On July 12th, Fig. 3 shows several sensitivity drops, but only some of them correspond to increases of PSTAB signals.
On July 13th, Figs. 4, 5 show some drops, but very few correlate with PSTAB. Instead, Fig. 6 shows numerous drops in good correspondence with PSTAB increased noise.
Finally, on July 14th, Figs. 7, 8, 9 display no evident correlation between the sensitivity drops and PSTAB issues. Note that the drop occurring at about 7:00 UTC is due to seismic noise, as mentioned in the operator report and in entry 46361.
Both unlocks reported in the afternoon shift report indeed originate from the INJ subsystem, with slight differences whose relevance should be gauged by experts.
The unlock at 14:57:04 UTC ( Fig. 1 ) is preceded by a visible increase of INJ_EOM_CORR, INJ_IMC_REFL signals ( Fig. 2 ), lasting about 1s before the final fireworks ( Fig. 3 )
The behaviour is different for the unlock at 19:57:16 UTC ( Fig. 4 ), for which there is no evident increase of signal levels before the unlock ( Fig. 5 ), whose final spasms are visible in Fig. 6.
ITF locked at 14:04UTC, Hrec signal was flat.
14:52UTC, ITF lost the lock (fast unlock, 1st plot)
15:19UTC, ITF locked again
15:48UTC, Science mode with squeezer engaged. Hrec Range BNS 42Mpc
19:57UTC ITF lost the lock (fast unlock, 2nd plot)
20:12UTC, first relock attempt failed at LN1, probabily due to SSFS low gain (3rd plot)
20:44UTC, relocked and set in Science, Hrec Range BNS 42Mpc
Guard Tours
13:21UTC - 13:48UTC
15:06UTC - 15:30UTC
17:31UTC - 18:03UTC
19:34UTC - 20:06UTC
SQZ
15:33 - 15:48UTC
Adjusting - Squeezer checking (entry #46363).
ISC
(12-07-2019 12:00 - ) contact the ISC on-call to align the BS
DAQ
(14-07-2019 14:15 - 14-07-2019 15:45) From remote
Status: Ended
Description: Hrec signal flat
Actions undertaken: Hrec signal reappeared next lock, no actions needed
After proper modification of the SQZ metatron node by Marco (more on this later in a separate entry), today after the Earthquake we tested the squeezing injection, first manually at 15:19 UTC, then with automation at 15:43.
We can reach LN3_SQZ smoothly.
We measure about 2.8 dB of squeezing, see attached plot.
According to data from the IRIS website (Event Plots section, see also the query interface) the waves from this event occurring in Indonesia at 9:10:50 UTC should have first arrived in Italy (about 110° away on Earth sphere) around 20 minutes later, at about 9:30 UTC.
This projection is well confirmed by the extra noise on the Virgo seismometers, as plotted in Fig. 1, with strong shaking starting right at 9:30 UTC and lasting for over one hour, clearly leading to the unlock of 9:41 UTC.
Good job anyway, withstanding such an event for 10 minutes!
According to data from the IRIS website (Event Plots section, see also the query interface) the waves from this event occurring in WA at 5:39:24 UTC should have first arrived in Italy (about 110° away on Earth sphere) around 15-20 minutes later, at about 5:55 - 6:00 UTC.
I see indeed evidence of extra noise on the Virgo seismometers, as plotted in Fig. 1, starting around 6:00 UTC, with a peak right at 5:55 UTC in correspondence of a drop of sensitivity. Later, one can see a relevant increase in seismic noise which may have a role in the sharp sensitivity drop occurring at 07:02:14 UTC.
P.S. Apologies for posting an earlier version of this report with incorrect conclusions, due to a blunder with my settings of dataDisplay time windows.
from 5:10UTC to 5:12UTC: discharge the 50Hz feed-forward
from 6:30UTC to 7:26UTC: switch the reallocation to the input mirrors trying to survive to this earthquake:
M 6.6 - 202km W of Broome, Australia
05:39:24 (UTC)
at 9:33UTC: switch the reallocation to the input mirrors trying to survive to this earthquake:
M 7.3 - 102km NNE of Laiwui, Indonesia
09:10:50 (UTC)
this one was too strong and the ITF unlocked at 9:41UTC; others earthquakes in the same region followed the main one and they prevented to relock for about two hours, at ~11:35UTC I started to relock the ITF, at the beginning the ITF failed at LOW_NOISE_1 state: some of them because of the alignment some others because of the SSFS boost, finally the ITF locked at LN1 and now we are waiting to warm it up before going on with the lock acquisition...
unfortunately the ITF unlocked because of a fast unlock with the injection going through a crisis, so I tried to kept the IMC not locked for a while and then go on, relock in progress...
-guard tours(UTC)
7:00 --> 7:25
8:55 --> 9:25
11:25 --> 11:55
DAQ
InfraMoniAlarmed stopped/restarted at 7:42UTC (threshold updated).
ISC
(12-07-2019 12:00 - ) contact the ISC on-call to align the BS
The unlock at 09:41:25 UTC, July 14th, is correlated with a single glitch in the INJ signals, reflecting in a glitch in the SSFS, which unlocks at 09:41:23.7439, see Figs 1, 2, 3.
Starting at 21:54 UTC, the BNS range went down to ~0.2 Mpc in ~20 seconds. It stayed around that value until 21:57:30 UTC when it went down to exactly 0 when the Hrec quality flags became active: Hrec_Flag_Quality went from 1 to -32 and then a few seconds later to -96 (= -32 -64 meaning that some bits were set). In parallel the state vector bits 0, 3, 4 and 10 became inactive, as shown in https://vim-online.virgo-gw.eu/resources/2019-07-13/resources/20190713_sum_statevector2d.png. From that time onwards the data were tagged as unusable for online analysis (bits 0 and 10 at 0). At the end of the bad period around 22:33:34 UTC the data quality flag went up to -64 and then to -1.The BNS range went back up at 22:33:46 UTC while state vector went finally back to its nominal state at 22:33:55 UTC (bits 1 and 2 had been inactive for a few seconds during the transition).
This night the ITF unlocked once at 3:08UTC. At 21:00 UTC I found the ITF locked with the Hrec Range BNS affected by many sensitivity drops, the ISC expert oncall was already alerted by Beatrice. Around 21:54 UTC, the Hrec BNS range dropped to zero however, the ITF stayed on lock, in meanwhile the PSTAB power stabilization signals saturated therefore, in agreement by the expert that was investigating the problem, I resetted the loop and the sensitivity had gone back to normal (22:33 UTC). After the unlock the lock acquisition has been difficult, Science mode achieved at 4:26UTC.
22:33UTC / 22:34UTC - ADJUSTING MODE - PSTAB has been opened/closed in order to remove the extra noise.
PSTAB
The PSTAB has been opened/closed in order to remove the extra noise. Action performed at UTC 22:33, 03:29, 03:48.
DMS
03:49 UTC - TFMoni crashed/restarted.
DMS/ACS
DMS showed thea INF__WEB_CHILLER_PRES is near to the threshold, expert informed by mail.
Guard Tour
22:31 - 23:01 UTC
00:10 - 00:40 UTC
02:06 - 02:37 UTC
04:05 - 04:30 UTC
ISC
(12-07-2019 12:00 - ) contact the ISC on-call to align the BS
ISC
(13-07-2019 20:30 - 14-07-2019 00:30) Operator on site with expert from remote
Status: Ended
Description: Lock affected by many deep sensitivity drops
Around 21:54 UTC, July 13th, the BNS range dropped to zero and the sensitivity update in vim-online stopped. However, the ITF stayed on lock, even though the spectra of hRec and DARM displayed a marked, stable increase in the broadband noise, as shown in Fig. 1. This drop was therefore different from previous sensitivity drops, which were temporary, lasting only a few tens of seconds.
The operator on phone confirmed that indeed the detector had not lost lock, and that upon a reset of the power stabilization loop the sensitivity had gone back to normal.
An inspection of the INJ signals shows that shortly before the sensitivity drop glitches are present in the PSTAB_PD1 signal (Fig. 2); one glitch is large enough to be seen as a glitch also in the PSTAB correction signal. Seconds later (Fig. 3) other glitches occur and apparently cause an unlock of the PSTAB loop, since the power stabilization signals saturate, shortly before 21:54:21 UTC.
In light of this episode, it is plausible that most of the sensitivity losses occurring today are due to similar issues with PSTAB. To be confirmed, by inspecting individual sensitivity loss events.
Starting at 21:54 UTC, the BNS range went down to ~0.2 Mpc in ~20 seconds. It stayed around that value until 21:57:30 UTC when it went down to exactly 0 when the Hrec quality flags became active: Hrec_Flag_Quality went from 1 to -32 and then a few seconds later to -96 (= -32 -64 meaning that some bits were set). In parallel the state vector bits 0, 3, 4 and 10 became inactive, as shown in https://vim-online.virgo-gw.eu/resources/2019-07-13/resources/20190713_sum_statevector2d.png. From that time onwards the data were tagged as unusable for online analysis (bits 0 and 10 at 0). At the end of the bad period around 22:33:34 UTC the data quality flag went up to -64 and then to -1.The BNS range went back up at 22:33:46 UTC while state vector went finally back to its nominal state at 22:33:55 UTC (bits 1 and 2 had been inactive for a few seconds during the transition).
The lock is affected by severe sensitivity drops, due to increases of broadband noise, like the one shown in Fig. 1, which last a few tens of seconds, as shown in the attached Movie 1.
Differently from the sensitivity drop occurring during the morning shift at ~10 UTC, which was correlated with an earthquake, for these drops there is no evidence of a correlation with excess seismic noise, see Fig. 2.
