2017-06-18-11h31m53-UTC>WARNING-AcChBufCheckStrGet> at least one cycle elapsed time 132967ns >= 100000ns loopPeriod at GPS1181820729-909312070
2017-06-18-11h31m53-UTC>WARNING-AcAdcChCheck> SSFS_B4_56MHz_Q - SSFS_B4_Err_Q_unnorm_10kHz delayed or missing from GPS1181820729-909410310 for more or less 0.998595(s) - nLoop 1@10000Hz
2017-06-18-11h31m53-UTC>WARNING-AcAdcChCheck> SSFS_B4_56MHz_I - SSFS_B4_Err_I_unnorm_10kHz delayed or missing from GPS1181820729-909410310 for more or less 0.998596(s) - nLoop 1@10000Hz
Looking at the channel LSC_Acl_elapsed_time, it seems that the critical value of 1e-4 seconds is exceeded about 30 times per hour, see fig 1. Zooming in on some random spike in the elapsed time (see fig 2), glitches can be seen in the corrections sent to at least the NE and WE mirrors, although this is not apparently visible in hrec.
Proposed actions:
-resolve the issue if possible.
-at the DSP side, add a probe on the correction received from LSC_Acl (I believe we had this, but I cannot find it anymore), possible with an _NS suffix. This can then be used to compute the difference between the (delayed) sent and received values, so that we can check for communication problems. We had channels like this in the past.
-check that these glitches were not affecting the sensitivity. It seems that the spikes in the elapsed time have been there for quite a while. I don't expect anything at low frequencies, but these glitches might for example excite violin and mirror modes, since they happen after the notches (which are located in the control filters in LSC_Acl).
In case of trouble, the switch in LSC_Acl.cfg, the reset commands at the bottom of LSC_UGF.cfg, and the switch-on of the switch in DOWN should all be commented.
Disabling the UGF did not help for the LSC_Acl elapsed time.
A new test: we stopped the ENVnoise process that runs on the same rtpc. It can be restarted if needed for noise injections, but let's try to keep it off when it is not used for some time.
As another test, we have modify ENVnoise so that it runs at 10 kHz on the same CPU as LSC_ACl and CALnoise, instead of running at 5 kHz on another CPU. The modification of packet frequency required to reload some processes on other rtpcs (SIB2, SNEW and SWEB SBE processes, CEB_TCS, ..). The modifications has been done between 13h20 and 13h38 UTC. After this modification, the processes on rtpc3 are running at either 10 kHz or 2 kHz, no more process at 5 kHz. However, there was still one channel generated in LSC_Acl at 5 kHz and provided to ENVnoise. This induce some sub-tasks at 5 kHz still being present.
At 16h30 UTC, we have modified/restarted LSC_Acl and ENVnoise so that the exchanged channel is also at 10 kHz. There should be nothing remaining at 5 kHz now.
The glitch is well visible in the spectrogram of LSC_DARM but not in the spectrogram of Hrec_hoft_20000Hz.
We can see many other glitches in the spectrogram of Hrec_hoft_20000Hz, which are typical low SNR glitches (SNR<8) visible by Omicron with a glitch rate
of about 1 per second. Elapsed_time glitches are a priori less annoying in h(t) than those low SNR glitches.
So, even if such glitches should be investigated and removed, they are not a problem for data analysis.
