This afternoon we found the DAQ in a strange state :

• several decrease of data flow without any reasons
• strange latency fluctuations

Loocking at the monitoring of the IRIGB signal at each building(zoom) one can see that:

• the WE signal disappears during the nigth: this was due to a failure on rtpc9-lk2, moving the optical transceiver on the rtpc9-lk1 allows to recover part of the data)
• the NE signal disappears around 9h00 UTC , with the channel available in the DAQ
• the MC signal disappears at the same period as the NE one
• Everithing remains OK in the CEB

As there was cabling activities in the DAQ for the network upgrade, we went in the DAQ room and we found that  the optical patch panel used to distribute the Timing signals at NE,WE,COB, and MC and for the Tolm network for the Suspenson in the same areas was disconnected ,

As consequence :

• the {Sa,Sc} datas for  the MC, NE and WE suspension were absent all the  day
• all the MC,NE,WE,COB Tolm devices have to be resynchronized

To recover  the correct running conditions, the missing fibers were reconnected and the INJ, NEB and WEB Tolm devices resynchronized

A the WEB the VME crate hosting the ADC7674 and the MxDx_v1 was found unpowered . Plugin back the cable allows to restart the VME crate