ITF found in Commissioning Mode, the ongoing activity ( Measurements of the charge of the test masses ) was completed at 13:10 UTC. From 13:14 UTC Carbognani and Boschi investigated the Communication problem with suspensions. Activity completed at 14:18 UTC, ITF switched to Science Mode.
At 19:01 UTC the ITF unlocked due to missing data of the WE Building, the issue was related to most of the signals of the building and different subsystem ( see attached plot #1 ). I went to the Building to investigate the situation, the various racks were powered so I contacted the DAQ Oncall to try to restore the nominal condition.
While working the racks suddenly switched off completely at around 20:30 UTC, while instead the lights of the building were still on. D'Andrea found that one of the UPS was broken due to a dead animal inside ( see attached image #2 ) and restored the other UPS.
Recovery in progress.
Electricity
(26-08-2019 19:13 - ) At 19:01 UTC the ITF unlocked due to missing data. This first event could be caused by a short blackout at the building, possibly showing a first failure of the UPS. From around 20:30 UTC the power of the racks went down, the issue was caused by a broken UPS ( see related oncall report ), but the nominal condition of all the devices of the WE Building had to be restored manually.
DAQ
(26-08-2019 19:05 - 26-08-2019 20:40) From remote
Status: Ended
Description: Missing Signals from WE Building.
Actions undertaken: I contacted Masserot to investigate the massive missing channels of the building and we started the recovery, but during these operations the UPS of the Building turned off, highlighting a different issue instead of a communication problem.
Electricity
(26-08-2019 19:20 - 26-08-2019 21:10) On site
Status: Ended
Description: WE Building UPS and generator not responding. Later the issue was identified as related to the UPS.
Actions undertaken: I first contacted the Oncall during the first investigation of the missing channels: the UPS and Generators of the WE were not available and they could be a possible cause of the problem. I went to the WE Building to verify the state but the current was present at that moment. At around 20:30 UTC the power of the racks went down, highlighting an electrical failure. D'Andrea came on site, verify the damage caused by a dead mouse inside one of the UPS and restored the other.