After the maintenance yesterday, around 18h LT Michal realized that there was something strange in the reconstructed h(t), in particular the cavity pole was around 260 Hz instead of 180 Hz. Also the optical gains were not as usual (see figures 1 and 2). On the contrary, the cavity pole estimated by ISC was still ok (see last figure).
After a lot of investigations, and some injections for measurements of the optical responses and for actuator responses, we found that the issue comes from the fact that the channels SAT_*_LN_P1 were all at 0, instead of being at 1, for BS, NE and WE (see figure 3).
We understood that the suspensions were in fact properly switched in their correct modes (by Metatron), but that the monitoring channels giving their modes were not correct. Hence Hrec was using the actuator models in HP for BS and LN1 for NE and WE, instead of LN1 for BS and LN2 for NE and WE. The use of incorrect actuator models directly impacts the estimation of the optical responses.
Note that the processes CoilsSb* were restarted around noon (see gray bands in figure 4) and the logs of these processes show a lot of error messages (python errors and ModBus errors), , to be checked if this is the source of the issue, or something else in the DSPs. We called Valerio Boschi around 23h30 LT about this issue, to be checked tomorrow morning.
Around 23h45 LT, we restarted Hrec by temporarily forcing the mode of the suspensions to their real mode, without using the SAT monitoring channels. The optical responses are now back to nominal values (see very last part of figure 5).
_____________________
Around 23h50 LT (21h50 UTC), we finally started the injections planned for this evening/night: injections of lines at high frequency (range 1 kHz to 8 kHz) with the NE and WE PCal, to re-estimate the mechanical response of the PCal due to excitation of mirror internal (drum) modes. (but we have skipped the injections for check hrec).
--> Injections should run to around 9h LT tomorrow morning.
___________________
To trigger this issue quickly next time, a flag must be added in the DMS to check that the suspensions are in the correct mode depending on the lock state.
