It unlocked at 16:37 UTC after 21h21m of continuous lock; the reason of the unlock seems due to missing data of many channels (see attached plot); looking at the log file of the process SDB_EDB_Fb we can see this kind of message:
2019-04-10-16h37m00-UTC>ERROR..-[TolmFrameBuilderFrontEnd] packet rejected for EDB_PC_DAQ_10000Hz, SDB1_FAST_SHUTTER_DAQ_10000Hz, SDB2_LC_DAQ_10000Hz, ...
To be confirmed by experts.
At 16:55 UTC the ITF was relocked at the first attempt with squeezer injected and PRCL->SSFS subtraction disabled because it had been left commented in ITF_LOCK.py.
At 16:57 I manually activated the PRCL->SSFS subtraction.
At 16:59 UTC ITF in Science mode.
ITF in Science mode at the end of the shift.
Guard tours (time in UTC)
from 19:40 to 20:30;
from 21:30 to 22:00
The ITF unlock at 20190410-16h37UTC is in coincidence with the lost of data by the SDB_EDB_Fb server .
Investigations inprogress
Looking at the various log-files, the OMCs think there is a problem with the temperature control and are shutdown, which 2 seconds later causes the ITF to unlock.
DET_MAIN looks whether the Fast Shutter process is working properly, it also detects the issue, but a few months ago we put a protection, which waits for 3seconds of consecutive data loss before reacting. The same could be implemented on the temperature control check for the OMC.
2019-04-10-16h36m59-UTC>WARNING-Temperature not controlled...
2019-04-10-16h37m00-UTC>INFO...-cm_send('SDB1_OMC', 'AcRelayChTranSet', 'OMC1_Peltier_stab_cmd_sw', -1)
2019-04-10-16h37m00-UTC>INFO...-cm_send('SDB1_OMC', 'AcRelayChTranSet', 'OMC2_Peltier_stab_cmd_sw', -1)
2019-04-10-16h37m00-UTC>INFO...-cm_send('SDB1_OMC', 'AcRampChTranSet', 'OMC1_T1_ramp', 1, -1)
2019-04-10-16h37m00-UTC>INFO...-cm_send('SDB1_OMC', 'AcRampChTranSet', 'OMC2_T1_ramp', 1, -1)
2019-04-10-16h37m00-UTC>WARNING-in SHUTDOWN main
2019-04-10-16h37m01-UTC>WARNING-in OMCs_UNLOCK main
