Attached plot, taken from VIM Hrec Moni, shows, on the upper part, the latency of Hrec and V1FromOnlineR (last step before h(t) is sent to online data analyses) and, on the lower part, the latency of FbmAlp (from which Hrec takes its input data) and PyDAQ and CALI (two inputs of FbmAlp which may be at the origin of some increase of latency that is observed on Hrec and V1FromOnlineR).
For PyDAQ, there is an increase of latency that occurs every day between 16h and 17h UTC. We can observe also, just for tonight, an increase around 21h UTC and a larger latency of PyDAQ between 22h and 05h...
For CALI, three latency glitches are visible. One of them may be linked to the injection that was done just before 20h UTC, the other two need to be investigated.
Looking more at the CALI latency spikes in the attached plot, I show the CALI, FbmMain and FDS_Img latencies around when the latency spikes happened. The first and the last spikes come from FbmMain and then FDS_Img. There was an issue with FDS_Img when these latency spikes happen when there are issues with FDS_Img. I have also attached the log of FDS_Img when the latency spike occurred.
The second spike is caused by the CALI node when it is in the DOWN state, from the log:
2024-05-11-19h45m38-UTC>INFO...-in DOWN main
2024-05-11-19h45m38-UTC>INFO...-DOWN> enable_actions: 1
2024-05-11-19h45m38-UTC>INFO...-RESET CALnoise process to its default configuration
2024-05-11-19h45m38-UTC>INFO...-cm_send(b'CALnoise', b'AcRelayChTranSet', b'CAL_filter_enbl', -1)
2024-05-11-19h45m38-UTC>INFO...-Waiting 2.0 second
2024-05-11-19h45m40-UTC>INFO...-cm_send(b'CALnoise', b'AcRelayChTranSet', b'PCAL_filter_enbl', -1)
2024-05-11-19h45m40-UTC>INFO...-Memory Used increase, cur. 2259476.00(KB), inc. 6156.00(KB)
2024-05-11-19h45m40-UTC>INFO...-Waiting 2.0 second
2024-05-11-19h45m42-UTC>INFO...-cm_send(b'CALnoise', b'ReloadConfig', b'', b'reset')
2024-05-11-19h45m42-UTC>INFO...-Waiting 1.0 second
The latency spike is caused by CALI's use of the virgotools.wait function in the CALI_Common library rather than self.timer[' XXX']. This causes the latency to spike for 5 seconds every time the CALI node goes into DOWN.
The CALI Metatron node has been modified this morning in order to prevent glitches in FbmAlp latency coming from calls to "wait" function in Metatron.
Instead of calling the function reset_CALnoise(), behind which there were waits for 2+2+1s , the basic actions to Acl are called directly, and timers are used to wait between the calls.
This has been modified in the following states: DOWN, CALIBRATED_* and CALIBRATION_FAILED* .
In the main function:
self.timer['RESET'] = 2
self.reset = False
log.info('DISABLE calibration filters')
disable_CAL_filters_computation('MIR')
disable_CAL_filters_computation('PCAL')
# reset_CALnoise()
In the run fuction (instead of a simple" return True" ) :
#wait until the timer duration is completed to request CALnoise reload
if self.timer['RESET']:
if self.reset == False:
log.info('RESET CALnoise process to its default configuration')
cm_send('CALnoise', 'ReloadConfig','','reset')
self.reset = True
return True #state reached: CALnoise reloaded after timer completed
else:
#state not yet reached: waiting for CALnoise reload to be requested
return False
Hopefully all the glitches from CALI node were coming from the calls to reset_CALnoise and should not be present anymore. To be checked in the coming days.
No side-effects are expected for the calibration injections, but this also needs to be confirmed.