Virgo Logbook

AdV-DAQ (On Call)

pacaud, berni, verkindt, masserot, carbognani, cortese - 15:26 Tuesday 17 May 2022 (55875)

olserver52 issues

This morning, at around 6h30 UTC, olserver52 started to misbehave, showing an unreliable ethernet connection. As the problem was not easy to fix, we decided to move all the DAQ processes running on this machine to the spare machine, olserver53.

Most of the other processes already running on olserver53 were left untouched, except SpectroMoni, EnvDataWrite and Spare_IN1, which were stopped to free some ressources.

The 50Hz processes fed by FbmFFE complained about missing data. That was due to outdated info files already present in /dev/shm/VirgoOnline/FbmFFE. A removal of these files and a restart of the processes fixed the issue.

InjMoni and DetMoni consistently crashed on restart. This was due to an invalid memory access from the Frame library. These processes are temporarily restarted using a MoniServer compiled against the latest Frame library which include a fix for this issue. Dider is preparing a new version of Moni to be installed in /virgoApp.

INJ_MAIN metatron instances also refused to start. The issue is not yet understood but we are suspecting this is the same issue as for the MoniServer instances. Franco is preparing a new metatron using the fixed Frame library.

Comments to this report:

pacaud - 15:38 Tuesday 17 May 2022 (55876)

At 2022-05-17 13h36m08 UTC, all the MoniServer instances were stoppped, and then restarted using Moni v2r11.

2022-05-17 13h36m08 UTC pacaud moniserver stopped (AliMoni, CaliMoni, HrecMoni not running and ignored)
2022-05-17 13h36m14 UTC pacaud process:DetMoni started
2022-05-17 13h36m18 UTC pacaud process:DaqMoni started
2022-05-17 13h36m19 UTC pacaud process:DQMoni started
2022-05-17 13h36m19 UTC pacaud process:EnvMoni started
2022-05-17 13h36m20 UTC pacaud process:InfraMoni started
2022-05-17 13h36m20 UTC pacaud process:InfraMoniAlarmed started
2022-05-17 13h36m21 UTC pacaud process:InjMoni started
2022-05-17 13h36m22 UTC pacaud process:ITFOnCallMoni started
2022-05-17 13h36m23 UTC pacaud process:QuadMoni started
2022-05-17 13h36m23 UTC pacaud process:ServersMoni started
2022-05-17 13h36m25 UTC pacaud process:SqzMoni started
2022-05-17 13h36m26 UTC pacaud process:SuspMoni started
2022-05-17 13h36m27 UTC pacaud process:SuspEBMoni started
2022-05-17 13h36m27 UTC pacaud process:SuspShortMoni started
2022-05-17 13h36m28 UTC pacaud process:TCSMoni started
2022-05-17 13h36m28 UTC pacaud process:VacuumMoni started
2022-05-17 13h36m49 UTC pacaud process:LockMoni started
2022-05-17 13h37m00 UTC pacaud process:SLCMoni started

bersanetti, carbognani - 19:54 Tuesday 17 May 2022 (55881)

The issue regarding INJ_MAIN was due to the read of problematic post-processing channels found in the full frame and possibly linked to the mix of Frame library versions in use.

The solution about the new Metatron version using the fixed Frame library will be pursued during the move to Python3; for the time being it was sufficient to use the FDIN_TAG facility (see #55487) in order to acquire from the frame only the needed and standard channels from the DAQ.

INJ_MAIN was then restarted correctly and ITF_LOCK, which was temporarily disconnected from its use, has been restored correctly.

Right now, the only nodes which still are missing the FDIN_TAG functionality are SQZ_MAIN, SQZ_FLT, TCS_MAIN and CALI.

bawaj - 22:47 Tuesday 17 May 2022 (55884)

SQZ_MAIN and SQZ_FLT nodes code was checked for the necessary channels and the FDIN_TAG was added. These processes shared Metatron.cfg configuration file between themselves and with the TCS_MAIN. To disentangle them, before placing the tag, the main configuration was edited to ensure separated cfg.

None of the nodes were turned on due to the on-going commissioning activity on the SQZ subsystem. The test of the tags will be performed in the next days.

bersanetti - 12:38 Wednesday 18 May 2022 (55894)

The TCS_MAIN Metatron node has been updated to use the FDIN_TAG functionality; possible issues will be addressed when the node will be in use again.

narnaud - 23:27 Wednesday 18 May 2022 (55901)

BRMSMon recompiled against Fd v8r43p3, as suggested by Benoît. New tag created (v6r0), VPM configuration updated and server restarted.

cortese - 13:41 Tuesday 07 June 2022 (56077)

The problem was that the old olserver52 server hardware has been turned on by mistake and therefore it was causing IP address and authentication membership contention with the current server.

The good news are that there are no native failures in the current olserver52 server and it can be used again in redundancy with olserver53