The attached plot show that the FbmAlp latency has been improved  thank to the new data collection path used for the PyDAQ frames,

The 2 first plots show the latency of the DAQ chain  from the FbmFFE frame merger to the FbmAlp one,  for the 2024-05-15-16h_17h-UTC and the  2024-05-16-16h_17h-UTC :

• the PyDAQ data collection is not on FbmAlp since the 2024-05-15,
• there is still some latency fluctuation during the 16h_17h daily period on the FbmAlp frame merger, in time coincidence with the ones on PyDAQ
• the maximal FbmAlp latency fluctuations  reach 3s while the ones at the FbmMain level reach only 0,9s

Searching for the culpit on FbmAlp frame providers, one can find some metatron nodes, mainly CALI and SQZ_FLT (see this plot)

The attached plot show the latency of the some FbmaAlp frame providers for the 2024-05-17-12-h-UTC to 2024-05-18-07h-UTC period

• the red rectangle refers to the FbmAlp latency jumps related to the SQZ_FLT Metatron server
• the orange rectangle refers to the FbmAlp latency jumps related to the CALI Metatron server
• the purple rectangle refers to the PyDAQ latency jumps in coincidence with the ones of the  SQZ_FLT Metatron server

This confirms the CALI and SQZ_FLT  servers as the  sources of the FbmAlp latency jumps

As trial to reduce the impact of the SQZ_FLT latency fluctuation on the FbmAlp one :

• the SQZ_FLT server is now sending its frames directly the the SqzMoni server , instead of FbmAlp previously
• the SQZ_FLT channels are now propagated through the Detector monitoring DAQ branch to be transmited to the FbmSt  frame merger
• The channel monitoring the SQZ_FLT latency is now the channel called Daq_SqzMoni_SQZ_FLT_latency  (see the plot)

Operation complete at 2024-05-21-10h15m-UTC

The attached plots show that

• thank the migration of the SQZ_FLT frame collection on the SqzMoni server , there is less glitches on the FbmAlp server  (purple and green rectangle)
• but few remains (red circle) without any time correlation with the PyUGF or SQZ_FLT latency glitches

All the latency glitches on FbmAlp server, related to the 2024-05-21-10h to 2025-05-22-10h time period have been analyzed

• for the 2024-05-21 10h to 14h time period, several lock acquisition sequences : latency glitches due to FbsDet and ITF_LOCK servers
• around 2024-05-21-15h30 : latency glicthes due the 2 FbsVac restarts
  • 2024-05-21-15h18m35-UTC    info sentenac     process:FbsVac started
  • 2024-05-21-15h28m01-UTC    info sentenac     process:FbsVac started
• around 2024-05-21-15h40: latency glicthes due t0 the ITF_LOCK server
• for the  2024-05-21  16h to 16h40 time period: latency glitches due the FbsDet (red rectangle), FbsAlp (orange rectangle) and SUS_PR node (yellow rectangle)
• around 2024-05-21-18h24: latency glicthes due to SUS nodes
• for the 2024-05-21 04h00 to 05h00 time period: latency glitches due the  SUS nodes (red rectangle), and one from FbmMain (green rectangle)

For  this time period analysis, over all these events only one is due to the FbmMain latency glitch , so as primary conclusion using FbmMain as frame provider for Hrec should help to reduce the number of latency glitches