The same issue occured again yesteday 20210124-21h58

In the attached plot one can see that when the issue occured on the rpc7 ;

• the slow frame builders without relation with some servers on the rpc7 have some troubles to collect theirs Slow monitoring servers
• the latency on the TolmFrameBuider increase during a short period of time
• Before the first issue one can see an smal increase and  some smal latency fluctuations on all the Tolm frame builder while before there was no fluctuations.These fluctuations are still here

Once more the SIB2 rtpc(rtpc7) is unreachable since 2021-01-30-23h37m45UTC.

Looking in more precisely  the rtpc7 state it appears that :

• the channels sent from to SIB2_rtpc(rtpc7) to the INJ_rtpc(rtpc19) are still correclty received :  no missing Tolm packets from the SIB2_rtpc on the INJ_rtpc . To be noted that the RFC is still locked
• the channels sent from to SIB2_rtpc(rtpc7) to the ISC_rtpc(rtpc20) are still correclty received :  no missing Tolm packets from the SIB2_rtpc on the ISC_rtpc

On  each rtpc there is 4 CPUs:

• the CPU0 is used for no real-time tasks as the TolmFrameBuilder and to handle the operating system operations (network, disk, ..)
• the CPU1, CPU2 and the  CPU3 are used only for realt-ime tasks. The CPU1 manages the I/O and the real-time task scheduling 

As the channels are still sent to 2  rtpcs and that theirs contents are not constant it seems that the RT-CPUs are still running , meaning that only the CPU0 is blocked.

The SIB2_rtpc was rebooted around 20210131-20h37UTC  then all the SIB2 servers were restarted . At the reboot time  the RFC unlocked and doesn't succeed to relock after,  even the RFC_6MHz_phi0 is the same as previously

The issue doesn't seem to be related to the RFC but to the Injection in general, as also the IMC unlocked and the corrections of the BPC froze, with a possible misalignment of the beam as seen from the IMC_Ref camera.

I put INJ_MAIN in DOWN.