Reports 1-1 of 1 Clear search Modify search
AdV-DAQ (Data collection)
masserot - 8:57 Saturday 20 April 2024 (64039) Print this report
DAQ - TolmFrameBuilder unable to send theirs frames

Around  2024-04-20-06h06m28 most of the TolmFrameBuidler servers were unable to transmit theirs frames : it s probably the cause of the ITF unlock

Below the message reported by some TolmFRameBuilder servers

  • ISC_Fb
    • 2024-04-20-06h06m28-UTC>ERROR..-frame 1397628405 not send to FbmFFE: sending queue is full
    • 2024-04-20-06h06m29-UTC>INFO...-output queue to FbmFFE has been flushed; Sending frame 1397628407
    • 2024-04-20-06h16m30-UTC>ERROR..-frame 1397629007 not send to FbmFFE: sending queue is full
    • 2024-04-20-06h16m31-UTC>INFO...-output queue to FbmFFE has been flushed; Sending frame 1397629009
  • SWEB_Fb
    • 2024-04-20-06h06m28-UTC>ERROR..-frame 1397628405 not send to FbmFFE: sending queue is full
    • 2024-04-20-06h06m29-UTC>INFO...-Memory Used increase, cur. 756764.00(KB), inc. 40.00(KB)
    • 2024-04-20-06h06m30-UTC>INFO...-output queue to FbmFFE has been flushed; Sending frame 1397628408
    • 2024-04-20-06h16m30-UTC>ERROR..-frame 1397629007 not send to FbmFFE: sending queue is full
  • SSFS_Fb
    • 2024-04-20-06h06m28-UTC>ERROR..-frame 1397628405 not send to FbmFFE: sending queue is full
    • 2024-04-20-06h06m29-UTC>INFO...-output queue to FbmFFE has been flushed; Sending frame 1397628407
    • 2024-04-20-06h16m30-UTC>ERROR..-frame 1397629007 not send to FbmFFE: sending queue is full
    • 2024-04-20-06h16m31-UTC>INFO...-output queue to FbmFFE has been flushed; Sending frame 1397629008
  • SUSP_Fb
    • 2024-04-20-06h06m28-UTC>ERROR..-frame 1397628405 not send to FbmFFE: sending queue is full
    • 2024-04-20-06h06m31-UTC>INFO...-output queue to FbmFFE has been flushed; Sending frame 1397628409
    • 2024-04-20-06h16m30-UTC>ERROR..-frame 1397629007 not send to FbmFFE: sending queue is full
    • 2024-04-20-06h16m33-UTC>INFO...-output queue to FbmFFE has been flushed; Sending frame 1397629011

The IT departement has been informed to check for a possible network switch issue

Comments to this report:
cortese, kraja, dibiase - 14:47 Monday 22 April 2024 (64059) Print this report

These events are likely correlated  to the burst of jobs that have been automatically submitted to the HTCondor farm by the DQR and detchar pipelines some time before, triggered by several H1 GstLAL alerts.

Between 4:20 UTC and 6:20 UTC about 1600 of these jobs have run causing a peak of more than 150MB/s nfs read traffic and consequent overload  of the fs01 fileserver where /virgoLog is hosted.

Since RTPCs still mount /virgo, /virgoApp and /cvmfs via NFS from fs01 instead of CVMFS, they may have been impacted by the corresponding slowdown which however has not provoked any visible errors at the Operating System level.

 

Search Help
×

Warning

×