Virgo Logbook

AdV-DAQ (On Call)

letendre - 23:42 Monday 11 October 2021 (53448)

SDB2_DBOX_LeftDown not reachable

At around 20:43 UTC, the DaqBox SDB2_DBOX_LeftDown (SN 11) was not reachable. This DaqBox was not responding on either the Tolm network or the Ethernet network.
A power OFF/ON cycle was performed in order to recover this DaqBox.
All SDB2 bench DaqBoxes were reconfigured. We had to reconfigured all these DaqBoxes because the timing is chained starting from SDB2_DBOX_LeftDown.

021-10-11 21h00m33 UTC letendre Disable power for device DboxLeftDown using PowerUnit09
2021-10-11 21h00m38 UTC letendre Disable power for device DboxLeftDown using PowerUnit09
2021-10-11 21h00m59 UTC letendre Enable power for device DboxLeftDown using PowerUnit09
2021-10-11 21h01m00 UTC letendre Enable power for device DboxLeftDown using PowerUnit09
2021-10-11 21h02m30 UTC letendre Reconfigure SDB2_DBOX_LeftDown in SDB2_dbox_bench
2021-10-11 21h03m21 UTC letendre Reconfigure SDB2_DBOX_LeftUp in SDB2_dbox_bench
2021-10-11 21h26m26 UTC letendre Reconfigure SDB2_DBOX_RightUp in SDB2_dbox_bench
2021-10-11 21h26m50 UTC letendre Reconfigure SDB2_DBOX_RightDown in SDB2_dbox_bench

Comments to this report:

pacaud - 14:14 Saturday 23 October 2021 (53619)

Same issue today.

2021-10-23-11h59m25-UTC    info pacaud       Session start
2021-10-23-12h02m14-UTC    info pacaud       Disable power for device DboxLeftDown using PowerUnit09
2021-10-23-12h02m25-UTC    info pacaud       Enable power for device DboxLeftDown using PowerUnit09
2021-10-23-12h02m45-UTC    info pacaud       Reconfigure SDB2_DBOX_LeftDown in SDB2_dbox_bench
2021-10-23-12h03m19-UTC    info pacaud       Reconfigure SDB2_DBOX_LeftUp in SDB2_dbox_bench
2021-10-23-12h03m52-UTC    info pacaud       Reconfigure SDB2_DBOX_RightUp in SDB2_dbox_bench
2021-10-23-12h04m47-UTC    info pacaud       Reconfigure SDB2_DBOX_RightDown in SDB2_dbox_bench

letendre - 19:37 Monday 25 October 2021 (53641)

This issue occurs when the DaqBox + 12V power supply drops below the undervoltage trigger.
This is due to the voltage drop in the power cables inside the minitowers when there is too much current drawn, mostly when shutters are activated (open or close actions)

Both SDB2 DaqBox crashes (11/10 and 23/10) are due to a "B1p_PD2 shutter open" command while shutter B5_PD2 was closing. B5_PD2 shutter was triggered on a hardware currant limit (20mA)

In the attached figures, SDB2_POWERSUPPLY_DBOX_LEFT_DOWN_p12V is the voltage at the output of the power supply. There is voltage compensation at the minitower flange to compensate for the voltage drop in the cables between the power supply and this minitower flange. There is no compensation inside the minitower. An increase in SDB2_POWERSUPPLY_DBOX_LEFT_DOWN_p12V means that there is more current in cables and then a voltage drop at the power input of the DaqBox.

Looking at B1p_PD2 shutter activity, one can see that a lot of "open B1p_PD2 shutter" commands are sent even if the shutter is already open. It would be better to check the shutter status before sending commands.

Images attached to this comment