The DAQ glitches can be explain by the fact that since 4 October the input flux to the fs01 nfs fileserver has increased again after the pause of September.

We have understood that the FdIOServer thread that sends frames via Cm is the same that writes logs on /virgoLog  (served by fs01) and therefore an overload of fs01 causes the whole thread to lag.

The increase of the flux (to some of the /virgoLog, /virgoData, /olusers or /virgoDev filesystems) happened again in correspondence of a restart of some virgo processes triggered by the unavailablity of the /data/archive area on 4 October.

From an analysis of the traffic on fs01 it turns out that the major writers are currently olserver119 and olserver114 in this order.

I see that on both server run some Fm processes which as far as I understand write on /virgoData , which stays indeed on fs01 , so explaining at least part of the overload.

It is not possible from the OS side to measure which of the processes on those 2 servers have a role because of the risk of impacting on the performances.

It should also be explained why the writing flux had been lower during all September.

As trial the FmRawBack server has been stopped at 2024-10-21-16h28-UTC

The FmRawBack server has been stopped at 15h28m-UTC (red line in the following plots)

The attached plots shows

• the effect of the fs1 input ethernet traffic: reduction of the input Ethernet flow
• that there is no effect on the latency jumps

As new trial, the following servers were restarted

• the Imaging servers running on the rtpcs
• the TolmFrameBuilder servers running on the rtpcs
• All the DAQ servers running on the olserver52 from FbmFFE  to FbmSt 
• Operation performed between  2024-10-22-06h35m32-UTC and  2024-10-22-06h38m39-UTC

Some plots showing  the evolution over 1 mont of some interesting channels 

• the fs1 Eternet data flow over 1 month - 2024-10-05 to 2024-1105
• the latency of the DAQ front end servers

One can see that even with the stop of the FmRawBack server and the restart of the DAQ front end servers, the glitches that appeared since the 2024-10-14-10h57-UTC are still present on the DAQ latency channels

Since the 2024-10-21-16h28-UTC , the FmRawBack server is stopped and none improvements were observed in the DAQ latency jumps 

The FmRawBack server, running on the olserver119 and providing the raw_bck.ffl file, has been restarted at  2024-10-05-07h52m34-UTC.

The FFL monitoring of the raw_bck.ffl has been restored  on the  FFLMoni3 server the 2024-11-16-05h37m10-UTC

The attached plots show the latency fluctuations for the TolmFrameBuilder servers running on the rtpc hosts  and the 2 first Frame Merging  servers running on the olserver52 host :

• from the  2024-10-02 to  the 2024-10-15
  • the latency glitches appeared the 2024-10-14-10h56m38s-UTC: zoom from 2024-10-12 to 2024-10-15
• from the 2024-10-16 to the 2024-10-29
• from the 2024-10-30 to the 2024-11-12
• from the 2024-11-13 to the 2024-11-26
  • the latency glitches disappeared the 2024-11-19 during the maintenance period without apparent reasons: zoom from the 2024-11-17 to 2024-11-20

In the meantime the FmRawBack server( 2024-10-05-07h52m34-UTC ) and the FFL monitoring  server ( 2024-11-16-05h37m10-UTC ) have been restored without obvious improvement in the glitch rate reduction

The origin of these glitches is not yet understood and remains unexplained, but they have disappeared as they appeared

During the maintenance window on Nov 19th we have performed some upgrades on the filservers' underlaying infrastructure ( Entry 65561 ).
This action may have helped on reducing the number of glitches, since it consisted on the upgrade of the drivers for the network cards and changing the policy of access to the storage, with the purpose to improve the latency.
Please note that even this action is in coincidence with the decrease of the glitches, no actions on the infrastructure were performed when the glitches started to appear on 14th of October.