Reports 1-1 of 1 Clear search Modify search
Virgo Runs (O3)
magazzu - 2:26 Tuesday 27 August 2019 (46775) Print this report
Operator Report - Afternoon shift

ITF found in Commissioning Mode, the ongoing activity ( Measurements of the charge of the test masses ) was completed at 13:10 UTC. From 13:14 UTC Carbognani and Boschi investigated the Communication problem with suspensions. Activity completed at 14:18 UTC, ITF switched to Science Mode.
At 19:01 UTC the ITF unlocked due to missing data of the WE Building, the issue was related to most of the signals of the building and different subsystem ( see attached plot #1 ). I went to the Building to investigate the situation, the various racks were powered so I contacted the DAQ Oncall to try to restore the nominal condition.
While working the racks suddenly switched off completely at around 20:30 UTC, while instead the lights of the building were still on. D'Andrea found that one of the UPS was broken due to a dead animal inside ( see attached image #2 ) and restored the other UPS.
Recovery in progress.

Downtime events

Electricity
(26-08-2019 19:13 - ) At 19:01 UTC the ITF unlocked due to missing data. This first event could be caused by a short blackout at the building, possibly showing a first failure of the UPS. From around 20:30 UTC the power of the racks went down, the issue was caused by a broken UPS ( see related oncall report ), but the nominal condition of all the devices of the WE Building had to be restored manually.

Oncall events

DAQ
(26-08-2019 19:05 - 26-08-2019 20:40) From remote
Status: Ended
Description: Missing Signals from WE Building.
Actions undertaken: I contacted Masserot to investigate the massive missing channels of the building and we started the recovery, but during these operations the UPS of the Building turned off, highlighting a different issue instead of a communication problem.

Electricity
(26-08-2019 19:20 - 26-08-2019 21:10) On site
Status: Ended
Description: WE Building UPS and generator not responding. Later the issue was identified as related to the UPS.
Actions undertaken: I first contacted the Oncall during the first investigation of the missing channels: the UPS and Generators of the WE were not available and they could be a possible cause of the problem. I went to the WE Building to verify the state but the current was present at that moment. At around 20:30 UTC the power of the racks went down, highlighting an electrical failure. D'Andrea came on site, verify the damage caused by a dead mouse inside one of the UPS and restored the other.

Images attached to this report
Comments to this report:
masserot - 10:33 Tuesday 27 August 2019 (46777) Print this report

I was called by the operator at  19h09UTC  to investigate the issue of missing data . I found the following situation:

  • No more data sent by the SWEB elements (TOLM and Ethernet networks)
  • No more data sent by the 2 ADC7674 in WEB
  • No more data sent by the Sa_WE and Sc_WE
  • No more monitoriing report form the SWEB power supplies.
  • The Daq boxes located outside of the SWEB were reachable and  running but the SWEB_DBOX_LC one (DBOX_SN43)  rusing the 10MHz mode had a very high timing_error

With the help of the operator in the WEB,  we checked that:

  • the 2 ADC7674 were reachable and we succeeded to reconfigure them
  • the VME MxDx used by the Long Suspension was correcly running by checking the optical transceiver LED status (TX and RX green with some ones blinking): meaning that  nothing was broken
  • The SWEB power supplies, the Hameg one, were ON but not reachable from the Ethernet network . A power-off and power-on cycle allows to recover them and then we were able to switch ON the SWEB elements and to reconfigure the Daq Boxes.

Unfortunately  a second  electrical  UPS shutdown occurred , as consequence  we decided to wait for the electrical expert to fixe the issue before to restore the working DAQ .

The WEB and SWEB DAQ were restored around  21h40UTC excepted for the WEB PCAL as the laser was probably in safety state .

Search Help
×

Warning

×