In parallel Eric and others completed the swap of the master laser.
In the late afternoon we relocked in LOCKED_RECOMBINED but after an unlock (17:40 UTC) the automation was stuck/crazy and the problem was not so evident.
At the beginning I thought a problem related with LNFS because the IMC had some troubles to relock and INJ_MAIN was not able to perform fmod_err. Moreover the last message on the log was about LNFS.
I contacted Eric, I went on the "Piscina" and we verified that the LNFS generator was working properly.
At the end I discovered that the problem was caused by DET_MAIN that it was in "unknown" state; to restore the problem I put the node in INIT.
We relocked again in LOCKED_RECOMBINED but I collected again the same problem; I went in touch with Julia and we decided to leave the ITF in LOCKED_ARMS.
INF
stop and restart of the eurothermServer at 14:00 UTC to restore the connection with WE module.
ISC
At ~20:24 UTC LSC_Acl crashed; it was restarted at the first attempt. Alain is investigating on it.
PENDING
- "UTA DET" still in "Portata Nominale"
- the large valve between BS and SR still closed
- ITF_LOCK still modified at the following row: 828->829, 1244->1245, 1650->1651.
The automation issue could be due to the fact that DET_MAIN is asked to go to SHUTTER_CLOSED in order to have ITF_LOCK reach LOCKED_RECOMBINED; looking at the log it seems that this failed with the following error:
2018-10-16-18h27m51-UTC>WARNING-[SHUTTER_CHECK_OPEN.run] USERMSG 0: No power on B1s1/Bs2 PDs, shutter probably not open
My naive guess is that something has changed after the B1s1/B5 swap.
About the crash of LSC_Acl, this is most probably the same problem already occurred in the past when, because of some weird interaction between Acl / cm / PythonVirgoTools, an Acl process crashes when something tries to access a recently modified ACL_SUM_CH or ACL_MATRIX_CH object, whose number of inputs has changed; in this case I think this was triggered by the removal on the B1p_119MHz signals from the LSC sensing matrices. I am not sure though how to cure the problem, as reloading a Metatron node is not always sufficient (but sometimes it is), as the AclServer objects in PythonVirgoTools do not completely replace themselves upon RELOAD. We could think about some "flush" directive which deletes the Python objects from the userspace before reloading a new instance...
In either case, the DET_MAIN slow shutter automation cannot decide what is the state of the slow shutter, as some power on B1p_PD2 is required for that test. So it correctly goes into the UNKNOWN state.
This night I received an alarm, LSC_Acl crashed again at 3:11 UTC, I restarted it properly at 3:14 UTC.