A few more plots about C9.

The unlocks due to DARM oscillating at 27 Hz are caused by the fact that lately, even if OMC1 locks at the same B1s2 power ~ 1.5 mW, the intercalibration between RF and DC readout has changed and DARM falls down to the lower end of the stability region and sometimes unlocks at this stage. This has been cured for the time being by changing the DARM gain at the same time of the hand-off of the error signal; however, since the same change of the gain happens in the opposite way a bit earlier in the lock acquisition, it makes more sense to recalibrate the DC error signals, which should only make the final DARM gain to be changed and the DC readout lock acquisition smoother.

The duty cycly of the last C9 Run was a good 76%, despite the couple of issue during the weekend; it has been computed using the usual DQ_META_ITF_Mode between the first Science flag after the Calibration (GPS=1210728903) and the last one on Monday morning (GPS=1210920680).

the unlocks on may 19, from 16:42 to 18:20 UTC are due to the excitation of the EIB, as shown in the attached plot.

This morning we have made a preliminary analysis of the cause of the unlocks that took place during the weekend. We have excluded the period when Eric went to the laser lab due to the excitation of the suspended injection bench (see entry 41464) and the period of troubleshooting after the problem with the DAQ (see entry 41467) and the manual unlocks.

We have divided the unlocks in two groups, the ones occurred during the lock acquisition and the ones occured when in Low Noise 3.

This morning we made a "Fast Unlock" manually from Low Noise 3, in order to compare it with the ones of the weekend 12.18.32 UTC.

From the analysis we made it is interesting to remark that once in Low Noise 3 the most common cause of unlock comes from the IMC. Instead during the lock acquisition we have experienced a wide causes of unlocks (without counting the punctual problems mentioned at the beginning). One well understood is the high gain of the SSFS when reaching DF, we need to adjust better this transition.

We marked in yellow the unlocks that are not yet understood, we will keep on investigating. Also we will keep on studying the Fast unlocks to try to find a cause.

Here the time of the different GPS troubles that occured during the C9 run

2018-05-18-23h07m47-UTC>WARNING-AcCompCheckStrGet> GPS:1210720084 - Error GPS:1210720083-80 - error loop number 20000/10000(10000)
2018-05-18-23h08m03-UTC>WARNING-AcCompCheckStrGet> GPS:1210720100 - Error GPS:1210720099-50 - error loop number 20000/10000(10000)
2018-05-18-23h10m18-UTC>WARNING-AcCompCheckStrGet> GPS:1210720235 - Error GPS:1210720234-60 - error loop number 20000/10000(10000)

2018-05-19-22h50m48-UTC>WARNING-AcCompCheckStrGet> GPS:1210805465 - Error GPS:1210805464-110 - error loop number 20000/10000(10000)
2018-05-19-22h51m03-UTC>WARNING-AcCompCheckStrGet> GPS:1210805480 - Error GPS:1210805479-90 - error loop number 20000/10000(10000)
2018-05-19-22h51m23-UTC>WARNING-AcCompCheckStrGet> GPS:1210805500 - Error GPS:1210805499-70 - error loop number 20000/10000(10000)
2018-05-19-23h02m14-UTC>WARNING-AcCompCheckStrGet> GPS:1210806151 - Error GPS:1210806150-40 - error loop number 20000/10000(10000)

2018-05-20-22h47m00-UTC>WARNING-AcCompCheckStrGet> GPS:1210891637 - Error GPS:1210891636-110 - error loop number 20000/10000(10000)
2018-05-20-22h47m18-UTC>WARNING-AcCompCheckStrGet> GPS:1210891655 - Error GPS:1210891654-100 - error loop number 20000/10000(10000)
2018-05-20-22h47m34-UTC>WARNING-AcCompCheckStrGet> GPS:1210891671 - Error GPS:1210891670-100 - error loop number 20000/10000(10000)
2018-05-20-22h59m12-UTC>WARNING-AcCompCheckStrGet> GPS:1210892369 - Error GPS:1210892368-50 - error loop number 20000/10000(10000)
2018-05-20-23h00m05-UTC>WARNING-AcCompCheckStrGet> GPS:1210892422 - Error GPS:1210892421-110 - error loop number 20000/10000(10000)
2018-05-20-23h02m23-UTC>WARNING-AcCompCheckStrGet> GPS:1210892560 - Error GPS:1210892559-70 - error loop number 20000/10000(10000)
 

This night the ITF unlocked twice and the relock was quite easy. It has been left locked at 7:45 UTC (more than 5 hours of continuous lock).

ITF events:

• 22.47 UTC ITF unlocked;
• 23:18 UTC ITF in SCIENCE mode;
• 00.25 UTC ITF unlocked;
• 00:42 UTC ITF in SCIENCE mode;

Guard tours

21:24 - 21:57 UTC
23:50 - 00:25 UTC
02:03 - 02:37 UTC
04:00 - 04:32 UTC
 

Today, I found the ITF in SCIENCE mode.

ITF events:

• 15.02 UTC ITF unlocked maybe because of the bad weather (heavy rain and strong wind)
• 15.41 UTC ITF Locked in LOW_NOISE_3
• 15.44 UTC ITF in SCIENCE mode
• 20.40 UTC ITF unlocked
• 20.56 UTC IITF Locked in LOW_NOISE_3
• 21.00 UTC ITF in SCIENCE mode

Vpm

15.29 UTC CoilSbWE restarted after a crash

Guard tours

• 15.05-15.48 UTC
• 16.58-17.33 UTC
• 18.55-19.29 UTC

ITF events:
11:35 UTC, ITF unlocked
11:49 UTC, first attempt failed at step 'LOCKING_PRITF_DF'
12:02 UTC, ITF in 'LOW_NOISE_3'
12:24 UTC, ITF in Science mode with Horizon about 16Mpc
To set the Science Mode I waited for more because the Horizon was very low at the beginning.

VPM DAQ
09:37 UTC - FbmMainUsers process restarted after a crash

Guard Tour started at 12:52UTC

Upon arrival I found ITF locked in Science Mode. Horizon 17.5 Mpc.

21:27 UTC: ITF unlocked for unknown reason. During the relocking sequence OMC1 didn'work, I found manually the temperature (sending the temperature of the previous lock looking the trend data) to ingage the PZT_out via Temp_Stab command on VPM. It worked!
21:56 UTC: ITF again in Science Mode. Horizon 17 Mpc.
22:26 UTC: ITF unlocked for unknown reason. During the relocking sequence OMC2 didn'work, I found manually the temperature (sending the temperature of the previous lock looking the trend data) to ingage the PZT_out via Temp_Stab command on VPM. It worked!
22:44 UTC: ITF again in Science Mode. Horizon 17 Mpc.
22:51 UTC: ITF unlocked for unknown reason.
23:00 UTC: during the relock sequence corrupted signals were sent by LSC to PR, BS, NI and WI.
23:00 UTC: Troubleshooting Mode
23:30 UTC: thanks to V. Boschi from remote we solved the problem restarting LSC_Acl process.
23:35 UTC: I tried to relock but NI was misaligned. I realigned it properly but ITF unlocked again.
23:40 UTC: I tried to relock but NE was misaligned. I realigned it properly but ITF unlocked again.
23:45 UTC: LOCKED_ARMS finally reached.
00:00 UTC: ITF unlocked systematically at LOCKING_CARM_TO_MC (3 attempts).
00:20 UTC: ISC OnCall contacted. During the expert investigation, I tried again but ITF unlocked systematically at LOCKING_CARM_TO_MC.
02:08 UTC: D. Bersanetti discovered a bug and restarted SSFS_Ctrl, ASC_Acl, SNEB_Photodiodes, SWEB_Photodiodes, SDB1_OMC, ISYS_Acl processes. Problem fixed.
02:10 UTC: I tried to relock but NI and NE was misaligned. I realigned it properly.
02:52 UTC: OMC1 failure, SDB1_OMC and ISYS_Acl restarted from remote bu D. Bersanetti.
03:00 UTC: ITF unstable, Unlocks at various steps.
03:18 UTC: ITF again Locked in LOW_NOISE_3.
03:21 UTC: Science Mode
Horizon decreased from 18 to 16 Mpc.

DMS - DAQ-Computing
Timing - CEB_DBOX_SSFS_timing_error went out of range the trend slow down since a week reaching -1000. Expert informed.

Guard Tour
20:35 - 21:07 UTC
23:00 - 23:34 UTC
01:15 - 01:48 UTC
03:10 - 03:44 UTC
04:32  UTC

The problem is more widespread, and it is related to the GPS issue some Acl servers have been suffering lately; the processes were stuck logging the same sequence each second (see attachment) and not receiving any command.

I restarted SSFS_Ctrl, ASC_Acl, SNEB_Photodiodes, SWEB_Photodiodes, SDB1_OMC, ISYS_Acl; after the restart the issue was not present anymore.

The ITF was locked again in LOW_NOISE_3.

Side notes:

• since before the beginning of the run the sidebands power SPRB_B4_112MHz_mag is much lower (~60%) than before (see figure), but the stability of the alignment loops is the same, so the other signals, so its seems just rescaled; this seems to have happened after a restart of the SPRB_Photodiodes process on Friday afternoon;
• i found that the dampers MIR_Z_ON are off on both the NI and WI mirrors, I don't remember if this is expected; the North Cavity in particular is moving quite a lot after an unlock.

After an unlock, corrupted signals were sent by LSC to PR, BS, NI and WI (see figure) mirror DSPs on every relock try. We solved the issue by restarting LSC_Acl process. Relocking is now in progress.

Today, I found the ITF in SCIENCE mode.

ITF events:

B.Montanari at the end of her shift contacted A.Pasqualetti for a problem with the communication of the vacuum WI rack. He came on site during my shift in order to restore it. According to the Commissioning Coordinator during this work of Antonio (from 13.17 UTC to 13.22 UTC) I put the ITF in "TROUBLESHOOTING". The problem was easily solved by Antonio and the operation do not cause any unlock.

    13.22 UTC Communication restored and ITF back to SCIENCE mode
    14:35 UTC, ITF unlock
    15.06 UTC LOW_NOISE 3
    15.08 UTC ITF in SCIENCE mode
    16.43 UTC ITF unlock

Then they were problems with the locking because the EIB_SBE was exited. 17.51 UTC I called the expert E.Genin that came on site at 18.09 UTC. We went together in the LB for fixing this issue that we properly solved. 18.30 UTC we were back in the Control Room and 18.50 UTC we were able to reach LOW_NOISE 3

    18.53 ITF in Science mode

    21.00 UTC ITF left in Science mode

Guard tour

    15.13-15.44 UTC
    1616-16.48 UTC
    18.29-19.00 UTC
  
OnCall

E.Genin and A.Pasqualetti came on site

DMS

20.58 UTC INF_WEB_TE_OUT threshold updated as requested by D.Soldani. InfraMoniAlarmed stopped and restarted.

ITF events:
- 08:32 UTC, ITF unlock
- 08:42 UTC, first attempt failed at step 'LOW_NOISE_1'
- 09:10UTC, I contacted the DET oncall because there is no ramp for the OMC1 temp stab. In according with R.Goauty on thelephone, I manually go in DOWN to repeat the locking acquisition and try to solve the issue. Seems a problem of Metatron.
- 09:19 UTC, lock failed at step 'LOCKING_PRITF_DF'
- 09:25 UTC, problem with the OMC1 locking caused by Metatron again, I contacted the ISC on-call to solve the issue. Under indication of D. Bersatti I performed the following procedure:
          - Set the node 'ITF_LOCK' in 'PAUSE';
          - Open the node 'OMC_LOCK';
          - Request 'OMCs_UNLOCKED' and wait that the node go in state 'TEMP_CONTROLLED';
          - Request 'OMC1_ONLY_LOCKED';
          - When the ramp restart, set ITF_LOCK in 'EXEC'
- 09:45 UTC, ITF in LOW_NOISE_3
- 09:50 UTC, ITF in Science mode with Horizon about 17 Mpc
- 11:52 UTC, ITF unlock
- 12:37 UTC, ITF in 'LOW_NOISE_3'
- 12:42 UTC, ITF in Science mode with Horizon about 17 Mpc

There are some problems with Metatron also at the beginning of the locking acquisition, it not restart automatically when the ITF go in down.

VPM - SAT
10:13UTC - CoilsSbWI process restart after a crash

Vacuum
No signal from the Cryotrap WI, in the afternoon will arrived the expert on site to restore the situation.

Guard Tour 12:00 - 12:50 UTC