First plot show the time constant of the event: 5 us. The PMC by itself is not able to to provide such fast actuation (pzt limited bandwidth to ~10 kHz, loop UGF ~few kHz).

PSL_ML_Freq_I_MONIT_FS is th independant meas. of the ML freq noise. It seems it is the one which triggers the event, the EOM follows in ordr to keep the IMC locked until it runs into saturation.

Although to be more conslusive one should put some numbers in Hz on the event dynamic it seems a reasonable hypothesis

As a partial integration of what already reported by Frederic on this event, we can see that at least the first lock-loss of the PMC at the beginning of the troubles was triggered by a loss of data from the BsX card providing (also) the corrections for the frequency loop. This was already reported by Alain in 45918.

For the record, the frequency monitor was not running at that time.

some more data on the PSL frequency behaviour
plot 1: shows the known signature of a fast unlock (EOM into saturation within 5 us)
the loss of data from the aquisition system seems to happen after this event (weird coincidence anyway)

plot 2: the same event, unzoomed. The ML_pzt and Thermal correction are clamped to zero as expected after the fast unlock. As a result we notice some counterpart on the ML_DC (ML power) 0,1-0,2 s later. The injection process of the slave resists to the unlocks and we can see that the SL_PZT is following the ML frequency.

plot 3: same event unzoomed. Here again, once the ML_TH control is clamped from 0 to 0,38 V a counterpart is visible in ML_DC and also SL_PZT_LF. The thermal effects within the ML crystal are likely to take place and drive some power perturbation, and a drift of the ML frequency (copied by the SL_pzt).

plot 4: a zoom on the relock. We see the 0,2 sec time constant between ML_PZT and (SL_PZT & ML_DC). The relock seems possible only after the proper data acquisition has been restored. Indeed from plot 3 we see that the PMC would have been able to relock many times since the fast unlock.

Event 2029-09-26-17h05m46UTC

• this plot shows 2 channels sent by the BsX DSP to the DAQ  where one can see that
  • the BxS data are unavailable during 5s
  • before being unavailable the channels content are filled to zero due to missing samples by the INJ_Fb TolmFrameBuilder
  • below the logfile content of the INJ_Fb frame builder, the one receiving the BxS DSP datas, related to  the previous items:
    • 2020-09-26-17h05m48-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 7944/10000 missing packet(s), 1 data break(s) . The Bxs DPS is expected to sent its Tolm packets at 10KHz, each packet contains one sample for each channel.

    • 2020-09-26-17h05m52-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1285175170.255704480
      2020-09-26-17h05m53-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 2557/10000 missing packet(s), 1 data break(s) . 

• this plot is a zoom  in the PMC unlock, where it appears clearly  that at the same time where the BxS DSP data  were set to zero by the INJ_Fb Tolm frame builder, the PMC and  the IMC  unlocks

Event 2020-09-27-10h40m05UTC

• This is roughly the same sequence, see the event and its zoom
  • BxS data  unavailable during 7s  and before being unavailable the channels content are filled to zero due to missing samples by the INJ_Fb TolmFrameBuilder
  • below the logfile content of the INJ_Fb frame builder

    • 2020-09-27-10h40m07-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 558/10000 missing packet(s), 1 data break(s)

    • 2020-09-27-10h40m12-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1285238430.009304500
      2020-09-27-10h40m13-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 93/10000 missing packet(s), 1 data break(s)

Event 2020-10-04-08h27m51UTC

This is roughly again  the same sequence, see the event and its zoom

• the logfile content of the INJ_Fb frame builder

  • 2020-10-04-08h27m53-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 340/10000 missing packet(s), 1 data break(s)

• 2020-10-04-08h27m58-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1285835296.062504480
  2020-10-04-08h27m59-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 625/10000 missing packet(s), 1 data break(s)

The last plot shows the trend over the last 2 weeks for the IMC and PMC unlocks and some BxS channels where the BxS missing data periods are shown by the grey lines: it seems that most of the PMC and IMC unlocks are appeared at the same time period of  BxS DSP malfunctioning  .

The file gives all the date of the missing samples from the BxS DPS starting from 2020-09-25 to 2020-10-05 (today)

In the attempt to fix the recurring issue with BsX data loss mentioned by Alain, the related DSP was rebooted this morning at around 12:40 LT. We will monitor the behavior of the signals.

Since the last reboot  ogf the BsX DSP done the 20201006-10h40UTC ,  the same troubles occurred again  at least 2 times:

• 2020-10-07-13h03m00-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 9105/10000 missing packet(s), 1 data break(s)
• 2020-10-07-13h03m04-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286111002.130304460
• 2020-10-07-13h03m05-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 10000 missing packet(s), 1 data break(s)

• 2020-10-08-05h06m25-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 291/10000 missing packet(s), 1 data break(s)
• 2020-10-08-05h06m30-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286168808.039404510
• 2020-10-08-05h06m31-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 394/10000 missing packet(s), 1 data break(s)

with the same sequences:

• an uncompleted channels inside the last frame: 9105/10000 and 291/10000 missing samples
• no data during 4 or 5 seconds
• an uncompleted channels inside the first frame after their recovery :  291/10000 and 394/10000 missing samples

Could it be possible to check if this DSP board is well connected to the Timing 

• • by looking if the Timing cable is well plugged at both sides (DSP and Timing Distribution box )
  • by reporting as SMS channel the Timing error information as for the others Tolm devices

Does someone known the meaning of this 5-6 seconds without data:  reset cycle , DSP restart ?

BxS troubes this nigth : 3 events . No more comments that the ones already reported

Event 20201008-16h42m36UTC

2020-10-08-16h42m38-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 339/10000 missing packet(s), 1 data break(s)
2020-10-08-16h42m43-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286210581.013104490
2020-10-08-16h42m44-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 131/10000 missing packet(s), 1 data break(s)

Event 20201008-23h00m39UTC
2020-10-08-23h00m41-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 4382/10000 missing packet(s), 1 data break(s)
2020-10-08-23h00m46-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286233263.603904430
2020-10-08-23h00m46-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 6039/10000 missing packet(s), 1 data break(s)

Event 20201008-23h10m44UTC
2020-10-08-23h10m46-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 5879/10000 missing packet(s), 1 data break(s)
2020-10-08-23h10m51-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286233868.442804480
2020-10-08-23h10m51-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 4428/10000 missing packet(s), 1 data break(s)

I am a bit lost,

do we know by which channels the drops in BsX trigger unlocks of both IMC/PMC?

Could those events (mentionned in ,49607) be cause of the fast unlocks? Assuming that BsX hiccup for only few ms (not long enough to appear as grey data)?

Below the list of the data collection troubles related to the BxS DSPs occured this week-end:

Event 2020-10-09-07h35m40

2020-10-09-07h35m42-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 2831/10000 missing packet(s), 1 data break(s)
2020-10-09-07h35m47-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286264164.769704500
2020-10-09-07h35m47-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 7697/10000 missing packet(s), 1 data break(s)

Event 2020-10-09-07h43m40

2020-10-09-07h43m42-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 5632/10000 missing packet(s), 1 data break(s)
2020-10-09-07h43m47-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286264644.467804470
2020-10-09-07h43m47-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 4678/10000 missing packet(s), 1 data break(s)

Event 2020-10-10-04h36m24

2020-10-10-04h36m26-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 72/10000 missing packet(s), 1 data break(s)
2020-10-10-04h36m31-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286339809.047004510
2020-10-10-04h36m32-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 470/10000 missing packet(s), 1 data break(s)

Event 2020-10-12-00h29m13

2020-10-12-00h29m15-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 4567/10000 missing packet(s), 1 data break(s)
2020-10-12-00h29m20-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286497777.639904510
2020-10-12-00h29m20-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 6399/10000 missing packet(s), 1 data break(s)

Below the list of BxS events siince the last entry:

Event 2020-10-13-08h42m59s:  IMC unlocked and PMC locked - PMC unlocks

2020-10-13-08h43m01-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 6481/10000 missing packet(s), 1 data break(s)
2020-10-13-08h43m06-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286613803.448304490
2020-10-13-08h43m06-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 4483/10000 missing packet(s), 1 data break(s)

Event 2020-10-13-15h45m54s : IMC unlocked and PMC locked - PMC unlocks
2020-10-13-15h45m56-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 6780/10000 missing packet(s), 1 data break(s)
2020-10-13-15h46m01-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286639178.368304470
2020-10-13-15h46m01-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 3683/10000 missing packet(s), 1 data break(s)

Event 2020-10-14-15h34m05s - IMC and PMC unlocked
2020-10-14-15h34m07-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 1828/10000 missing packet(s), 1 data break(s)
2020-10-14-15h34m12-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286724869.927904500
2020-10-14-15h34m12-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 9279/10000 missing packet(s), 1 data break(s)

Event 2020-10-14-15h57m50s - IMC and PMC unlocked
2020-10-14-15h57m52-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 5180/10000 missing packet(s), 1 data break(s)
2020-10-14-15h57m57-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286726294.512804500
2020-10-14-15h57m57-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 5128/10000 missing packet(s), 1 data break(s)

Event 2020-10-15-00h54m29s: PMC and IMC locked - IMC unlocks
2020-10-15-00h54m31-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 2201/10000 missing packet(s), 1 data break(s)
2020-10-15-00h54m36-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286758493.843604510
2020-10-15-00h54m36-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 8436/10000 missing packet(s), 1 data break(s)

Event 2020-10-15-06h07m20s: PMC and IMC locked - PMC and IMC unlocks
2020-10-15-06h07m22-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 5128/10000 missing packet(s), 1 data break(s)
2020-10-15-06h07m27-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286777264.533904480
2020-10-15-06h07m27-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 5339/10000 missing packet(s), 1 data break(s)

Below the last BxS events detected by the DAQ

Event 2020-10-15-11h38m35-UTC

2020-10-15-11h38m37-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 2066/10000 missing packet(s), 1 data break(s)
2020-10-15-11h38m42-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286797139.833004490
2020-10-15-11h38m42-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 8330/10000 missing packet(s), 1 data break(s)

Event 2020-10-15-12h17m19UTC
2020-10-15-12h17m21-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 5232/10000 missing packet(s), 1 data break(s)
2020-10-15-12h17m26-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286799463.506004450
2020-10-15-12h17m26-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 5060/10000 missing packet(s), 1 data break(s)

Event 2020-10-15-12h39m31UTC
2020-10-15-12h39m33-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 7812/10000 missing packet(s), 1 data break(s)
2020-10-15-12h39m38-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286800795.251104510
2020-10-15-12h39m38-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 2511/10000 missing packet(s), 1 data break(s)

The Timing cable connected to master BPC DSP has been replaced this morning, with an equivalent shielded one.

Below the same event as previously : lost of samples, missing data during 4-5s, recovery)

2020-10-28-19h39m37-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 8069/10000 missing packet(s), 1 data break(s)
2020-10-28-19h39m42-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1287949199.238304510
2020-10-28-19h39m43-UTC>ERROR..-[TolmFrameBuilderSource::StoreData] producer BsX channel BsX_PWRTrig: substitution of missing data is over frame bound (3366 extra value(s)) (35 times)
2020-10-28-19h39m43-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 2383/10000 missing packet(s), 1 data break(s)
2020-10-28-19h39m44-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 8057/10000 missing packet(s), 1 data break(s)
2020-10-28-19h39m44-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 13366/10000 missing packet(s), 1 data break(s)
2020-10-28-19h39m45-UTC>ERROR..-[TolmFrameBuilderSource::StoreData] producer BsX channel BsX_PWRTrig: substitution of missing data is over frame bound (3410 extra value(s)) (35 times)
2020-10-28-19h39m46-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 13410/10000 missing packet(s), 1 data break(s)

An updated DSP boot image has been uploaded in BPC DSPs. The problem should be now fixed.

Since the update of the DSP BsX firmware, there is still some troubles with its data collection. Here the list of the different events:

• 2020-11-11-13h40m23-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 2491/10000 missing packet(s), 1 data break(s) : missing samples inside the last frame
  2020-11-11-13h40m23-UTC>INFO...-[TolmFrameBuilderSourceManager::Clean] Source BsX (version 5) has stopped -> removed :
  2020-11-11-13h40m28-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 7873/10000 missing packet(s), 1 data break(s)  missing samples inside the new frame with no data during 4-5s

  This event triggered some misfunctioning of the INJ_Fb server

• 2020-11-12-01h05m52-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 5222/10000 missing packet(s), 1 data break(s)
  2020-11-12-01h05m52-UTC>INFO...-[TolmFrameBuilderSourceManager::Clean] Source BsX (version 5) has stopped -> removed
  2020-11-12-01h05m57-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 5308/10000 missing packet(s), 1 data break(s)

• 2020-11-12-22h41m35-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 4479/10000 missing packet(s), 1 data break(s)
  2020-11-12-22h41m36-UTC>INFO...-[TolmFrameBuilderSourceManager::Clean] Source BsX (version 5) has stopped -> removed
  2020-11-12-22h41m40-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 6029/10000 missing packet(s), 1 data break(s)

• 2020-11-15-19h14m41-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 1436/10000 missing packet(s), 1 data break(s)
  2020-11-15-19h14m42-UTC>INFO...-[TolmFrameBuilderSourceManager::Clean] Source BsX (version 5) has stopped -> removed
  2020-11-15-19h14m46-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 9041/10000 missing packet(s), 1 data break(s)

• 2020-11-16-02h28m11-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 6115/10000 missing packet(s), 1 data break(s)
  2020-11-16-02h28m12-UTC>INFO...-[TolmFrameBuilderSourceManager::Clean] Source BsX (version 5) has stopped -> removed
  2020-11-16-02h28m16-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 4928/10000 missing packet(s), 1 data break(s)

The plot  taken with the raw data is related to this event 2020-11-12-01h05m51 . The channel is PSL_PMC_Tra_DC_MoniT is equal to the BsX_QF_DC BsX DSP channel readed by the ISYS_moni Acl server . Note that the  PSL_PMC_Tra_DC_MoniT became constante as the same time as the  BsX_PZT_DCH one became equal to zero .

Using the raw data stream it seems that there is some coincidences with the IMC unlock and the BsX DSP troubles

The IMC lock loss is due for sure to the fact that BPC board is not working for a while. From the attached plot, one can see that the beam control is lost, then IMC is quickly misaligned up to the lock loss. It is also visible an effect on PMC, because at the same time the thermal correction provided by one DAC channel of the same board is not available anymore. In normal condition this fact does not cause a lock loss of PMC (but it can happen, for example if the thermal correction is large). The effect of the BPC board malfunctioning is normal: the same things happened several times during O3, We cannot avoid the loss of beam control, if the control board stops working.

For who cannot click on the link, the linked entry is #46450.

The increase of power at the level of PMC_TRA is likely due to the PSTAB trying to correct the drop of power in IMC_TRA.

Unfortunately the DSP boot image version uploaded on November 10th was wrong and was not containing the fix . Yesterday at 16:40 LT we uploaded the correct version on the BPC DSPs  that should fix the DAQ communication problem.