Reports 1-1 of 1 Clear search Modify search
AdV-DAQ (Data collection)
verkindt - 9:48 Sunday 12 May 2024 (64233) Print this report
Glitches in Hrec latency

Attached plot, taken from VIM Hrec Moni, shows, on the upper part, the latency of Hrec and V1FromOnlineR (last step before h(t) is sent to online data analyses) and, on the lower part, the latency of FbmAlp (from which Hrec takes its input data) and PyDAQ and CALI (two inputs of FbmAlp which may be at the origin of some increase of latency that is observed on Hrec and V1FromOnlineR).
For PyDAQ, there is an increase of latency that occurs every day between 16h and 17h UTC. We can observe also, just for tonight, an increase around 21h UTC and a larger latency of PyDAQ between 22h and 05h...
For CALI, three latency glitches are visible. One of them may be linked to the injection that was done just before 20h UTC, the other two need to be investigated.

Images attached to this report
Comments to this report:
poulton - 14:20 Tuesday 21 May 2024 (64313) Print this report

Looking more at the CALI latency spikes in the attached plot, I show the CALI, FbmMain and FDS_Img latencies around when the latency spikes happened. The first and the last spikes come from FbmMain and then FDS_Img. There was an issue with FDS_Img when these latency spikes happen when there are issues with FDS_Img. I have also attached the log of FDS_Img when the latency spike occurred.

The second spike is caused by the CALI node when it is in the DOWN state, from the log:

2024-05-11-19h45m38-UTC>INFO...-in DOWN main
2024-05-11-19h45m38-UTC>INFO...-DOWN> enable_actions: 1
2024-05-11-19h45m38-UTC>INFO...-RESET CALnoise process to its default configuration
2024-05-11-19h45m38-UTC>INFO...-cm_send(b'CALnoise', b'AcRelayChTranSet', b'CAL_filter_enbl', -1)
2024-05-11-19h45m38-UTC>INFO...-Waiting 2.0 second
2024-05-11-19h45m40-UTC>INFO...-cm_send(b'CALnoise', b'AcRelayChTranSet', b'PCAL_filter_enbl', -1)
2024-05-11-19h45m40-UTC>INFO...-Memory Used increase, cur. 2259476.00(KB), inc. 6156.00(KB)
2024-05-11-19h45m40-UTC>INFO...-Waiting 2.0 second
2024-05-11-19h45m42-UTC>INFO...-cm_send(b'CALnoise', b'ReloadConfig', b'', b'reset')
2024-05-11-19h45m42-UTC>INFO...-Waiting 1.0 second

The latency spike is caused by CALI's use of the virgotools.wait function in the CALI_Common library rather than self.timer[' XXX']. This causes the latency to spike for 5 seconds every time the CALI node goes into DOWN.

Images attached to this comment
Non-image files attached to this comment
rolland, masserot, verkindt - 13:14 Tuesday 28 May 2024 (64361) Print this report

The CALI Metatron node has been modified this morning in order to prevent glitches in FbmAlp latency coming from calls to "wait" function in Metatron.

Instead of calling the function reset_CALnoise(), behind which there were waits for 2+2+1s , the basic actions to Acl are called directly, and timers are used to wait between the calls.

This has been modified in the following states: DOWN,  CALIBRATED_* and CALIBRATION_FAILED* .

In the main function:

        #set timer to wait between filter relays disabling and reseting CALnoise configuration
        self.timer['RESET'] = 2
        self.reset = False
        # First disable all the relays for filter computations (and apply it immediately)'DISABLE calibration filters')
        # reset_CALnoise()

In the run fuction (instead of a simple" return True" ) :

    def run(self):
        #wait until the timer duration is completed to request CALnoise reload
        if self.timer['RESET']:
            if self.reset == False:
      'RESET CALnoise process to its default configuration')
                cm_send('CALnoise', 'ReloadConfig','','reset')
                self.reset = True
            return True #state reached: CALnoise reloaded after timer completed
            #state not yet reached: waiting for CALnoise reload to be requested
            return False

Hopefully all the glitches from CALI node were coming from the calls to reset_CALnoise and should not be present anymore. To be checked in the coming days.

No side-effects are expected for the calibration injections, but this also needs to be confirmed.

Search Help