Reports 1-1 of 1 Clear search Modify search
AdV-PSL (Pre-mode-cleaner block)
chiummo - 9:31 Monday 28 September 2020 (49517) Print this report
PMC lock troubles?

During the weekend som drops of power occurred to the IMC. Having a look at the upstream stages of the injection system, they seem related to some troubles at the PMC lock. In the attached plot, the power of the IMC follows the power at the output of the PMC, dropping in a couple of occasions while the neovan output power stays more or less constant.

The power drops seem occurring because the PMC looses its lock (see the error signals which deviates from its steady state value and fluctuates a lot during the drops, and the corrections on the PMC piezo).

To be investigated.

Images attached to this report
Comments to this report:
cleva - 7:59 Wednesday 30 September 2020 (49519) Print this report

First plot show the time constant of the event: 5 us. The PMC by itself is not able to to provide such fast actuation (pzt limited bandwidth to ~10 kHz, loop UGF ~few kHz).

PSL_ML_Freq_I_MONIT_FS is th independant meas. of the ML freq noise. It seems it is the one which triggers the event, the EOM follows in ordr to keep the IMC locked until it runs into saturation.

 

Although to be more conslusive one should put some numbers in Hz on the event dynamic it seems a reasonable hypothesis

 

 

Images attached to this comment
chiummo - 12:06 Sunday 04 October 2020 (49566) Print this report

As a partial integration of what already reported by Frederic on this event, we can see that at least the first lock-loss of the PMC at the beginning of the troubles was triggered by a loss of data from the BsX card providing (also) the corrections for the frequency loop. This was already reported by Alain in 45918.

For the record, the frequency monitor was not running at that time.

Images attached to this comment
cleva, chiummo - 10:38 Monday 05 October 2020 (49568) Print this report

some more data on the PSL frequency behaviour
plot 1: shows the known signature of a fast unlock (EOM into saturation within 5 us)
the loss of data from the aquisition system seems to happen after this event (weird coincidence anyway)

plot 2: the same event, unzoomed. The ML_pzt and Thermal correction are clamped to zero as expected after the fast unlock. As a result we notice some counterpart on the ML_DC (ML power) 0,1-0,2 s later. The injection process of the slave resists to the unlocks and we can see that the SL_PZT is following the ML frequency.

plot 3: same event unzoomed. Here again, once the ML_TH control is clamped from 0 to 0,38 V a counterpart is visible in ML_DC and also SL_PZT_LF. The thermal effects within the ML crystal are likely to take place and drive some power perturbation, and a drift of the ML frequency (copied by the SL_pzt).

plot 4: a zoom on the relock. We see the 0,2 sec time constant between ML_PZT and (SL_PZT & ML_DC). The relock seems possible only after the proper data acquisition has been restored. Indeed from plot 3 we see that the PMC would have been able to relock many times since the fast unlock.

 

Images attached to this comment
masserot - 12:28 Monday 05 October 2020 (49571) Print this report

Event 2029-09-26-17h05m46UTC

  • this plot shows 2 channels sent by the BsX DSP to the DAQ  where one can see that
    • the BxS data are unavailable during 5s
    • before being unavailable the channels content are filled to zero due to missing samples by the INJ_Fb TolmFrameBuilder
    • below the logfile content of the INJ_Fb frame builder, the one receiving the BxS DSP datas, related to  the previous items:
      • 2020-09-26-17h05m48-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 7944/10000 missing packet(s), 1 data break(s) . The Bxs DPS is expected to sent its Tolm packets at 10KHz, each packet contains one sample for each channel.
      • 2020-09-26-17h05m52-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1285175170.255704480
        2020-09-26-17h05m53-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 2557/10000 missing packet(s), 1 data break(s) . 

  • this plot is a zoom  in the PMC unlock, where it appears clearly  that at the same time where the BxS DSP data  were set to zero by the INJ_Fb Tolm frame builder, the PMC and  the IMC  unlocks

Event 2020-09-27-10h40m05UTC

  • This is roughly the same sequence, see the event and its zoom
    • BxS data  unavailable during 7s  and before being unavailable the channels content are filled to zero due to missing samples by the INJ_Fb TolmFrameBuilder
    • below the logfile content of the INJ_Fb frame builder
      • 2020-09-27-10h40m07-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 558/10000 missing packet(s), 1 data break(s)

      • 2020-09-27-10h40m12-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1285238430.009304500
        2020-09-27-10h40m13-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 93/10000 missing packet(s), 1 data break(s)

Event 2020-10-04-08h27m51UTC

This is roughly again  the same sequence, see the event and its zoom

  • the logfile content of the INJ_Fb frame builder
    • 2020-10-04-08h27m53-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 340/10000 missing packet(s), 1 data break(s)

  • 2020-10-04-08h27m58-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1285835296.062504480
    2020-10-04-08h27m59-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 625/10000 missing packet(s), 1 data break(s)

The last plot shows the trend over the last 2 weeks for the IMC and PMC unlocks and some BxS channels where the BxS missing data periods are shown by the grey lines: it seems that most of the PMC and IMC unlocks are appeared at the same time period of  BxS DSP malfunctioning  .

The file gives all the date of the missing samples from the BxS DPS starting from 2020-09-25 to 2020-10-05 (today)

Images attached to this comment
Non-image files attached to this comment
Boschi, chiummo - 14:28 Tuesday 06 October 2020 (49580) Print this report

In the attempt to fix the recurring issue with BsX data loss mentioned by Alain, the related DSP was rebooted this morning at around 12:40 LT. We will monitor the behavior of the signals.

Images attached to this comment
masserot - 12:14 Thursday 08 October 2020 (49599) Print this report

Since the last reboot  ogf the BsX DSP done the 20201006-10h40UTC ,  the same troubles occurred again  at least 2 times:

  • 2020-10-07-13h03m00-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 9105/10000 missing packet(s), 1 data break(s)
  • 2020-10-07-13h03m04-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286111002.130304460
  • 2020-10-07-13h03m05-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 10000 missing packet(s), 1 data break(s)

 

  • 2020-10-08-05h06m25-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 291/10000 missing packet(s), 1 data break(s)
  • 2020-10-08-05h06m30-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286168808.039404510
  • 2020-10-08-05h06m31-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 394/10000 missing packet(s), 1 data break(s)

 

with the same sequences:

  • an uncompleted channels inside the last frame: 9105/10000 and 291/10000 missing samples
  • no data during 4 or 5 seconds
  • an uncompleted channels inside the first frame after their recovery :  291/10000 and 394/10000 missing samples

Could it be possible to check if this DSP board is well connected to the Timing 

    • by looking if the Timing cable is well plugged at both sides (DSP and Timing Distribution box )
    • by reporting as SMS channel the Timing error information as for the others Tolm devices

Does someone known the meaning of this 5-6 seconds without data:  reset cycle , DSP restart ?

masserot - 7:47 Friday 09 October 2020 (49607) Print this report

BxS troubes this nigth : 3 events . No more comments that the ones already reported

Event 20201008-16h42m36UTC

2020-10-08-16h42m38-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 339/10000 missing packet(s), 1 data break(s)
2020-10-08-16h42m43-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286210581.013104490
2020-10-08-16h42m44-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 131/10000 missing packet(s), 1 data break(s)

Event 20201008-23h00m39UTC
2020-10-08-23h00m41-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 4382/10000 missing packet(s), 1 data break(s)
2020-10-08-23h00m46-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286233263.603904430
2020-10-08-23h00m46-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 6039/10000 missing packet(s), 1 data break(s)

Event 20201008-23h10m44UTC
2020-10-08-23h10m46-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 5879/10000 missing packet(s), 1 data break(s)
2020-10-08-23h10m51-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286233868.442804480
2020-10-08-23h10m51-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 4428/10000 missing packet(s), 1 data break(s)

Images attached to this comment
cleva - 8:25 Friday 09 October 2020 (49609) Print this report

I am a bit lost,

do we know by which channels the drops in BsX trigger unlocks of both IMC/PMC?

Could those events (mentionned in ,49607) be cause of the fast unlocks? Assuming that BsX hiccup for only few ms (not long enough to appear as grey data)?

masserot - 9:24 Monday 12 October 2020 (49625) Print this report

Below the list of the data collection troubles related to the BxS DSPs occured this week-end:

Event 2020-10-09-07h35m40

2020-10-09-07h35m42-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 2831/10000 missing packet(s), 1 data break(s)
2020-10-09-07h35m47-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286264164.769704500
2020-10-09-07h35m47-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 7697/10000 missing packet(s), 1 data break(s)

Event 2020-10-09-07h43m40

2020-10-09-07h43m42-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 5632/10000 missing packet(s), 1 data break(s)
2020-10-09-07h43m47-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286264644.467804470
2020-10-09-07h43m47-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 4678/10000 missing packet(s), 1 data break(s)


Event 2020-10-10-04h36m24

2020-10-10-04h36m26-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 72/10000 missing packet(s), 1 data break(s)
2020-10-10-04h36m31-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286339809.047004510
2020-10-10-04h36m32-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 470/10000 missing packet(s), 1 data break(s)

Event 2020-10-12-00h29m13

2020-10-12-00h29m15-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 4567/10000 missing packet(s), 1 data break(s)
2020-10-12-00h29m20-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286497777.639904510
2020-10-12-00h29m20-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 6399/10000 missing packet(s), 1 data break(s)

 

Images attached to this comment
masserot - 14:00 Thursday 15 October 2020 (49665) Print this report

Below the list of BxS events siince the last entry:

Event 2020-10-13-08h42m59s:  IMC unlocked and PMC locked - PMC unlocks

2020-10-13-08h43m01-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 6481/10000 missing packet(s), 1 data break(s)
2020-10-13-08h43m06-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286613803.448304490
2020-10-13-08h43m06-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 4483/10000 missing packet(s), 1 data break(s)

Event 2020-10-13-15h45m54s : IMC unlocked and PMC locked - PMC unlocks
2020-10-13-15h45m56-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 6780/10000 missing packet(s), 1 data break(s)
2020-10-13-15h46m01-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286639178.368304470
2020-10-13-15h46m01-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 3683/10000 missing packet(s), 1 data break(s)

Event 2020-10-14-15h34m05s - IMC and PMC unlocked
2020-10-14-15h34m07-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 1828/10000 missing packet(s), 1 data break(s)
2020-10-14-15h34m12-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286724869.927904500
2020-10-14-15h34m12-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 9279/10000 missing packet(s), 1 data break(s)

Event 2020-10-14-15h57m50s - IMC and PMC unlocked
2020-10-14-15h57m52-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 5180/10000 missing packet(s), 1 data break(s)
2020-10-14-15h57m57-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286726294.512804500
2020-10-14-15h57m57-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 5128/10000 missing packet(s), 1 data break(s)

Event 2020-10-15-00h54m29s: PMC and IMC locked - IMC unlocks
2020-10-15-00h54m31-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 2201/10000 missing packet(s), 1 data break(s)
2020-10-15-00h54m36-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286758493.843604510
2020-10-15-00h54m36-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 8436/10000 missing packet(s), 1 data break(s)

Event 2020-10-15-06h07m20s: PMC and IMC locked - PMC and IMC unlocks
2020-10-15-06h07m22-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 5128/10000 missing packet(s), 1 data break(s)
2020-10-15-06h07m27-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286777264.533904480
2020-10-15-06h07m27-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 5339/10000 missing packet(s), 1 data break(s)

 

Images attached to this comment
masserot - 9:29 Monday 19 October 2020 (49686) Print this report

Below the last BxS events detected by the DAQ

Event 2020-10-15-11h38m35-UTC

2020-10-15-11h38m37-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 2066/10000 missing packet(s), 1 data break(s)
2020-10-15-11h38m42-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286797139.833004490
2020-10-15-11h38m42-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 8330/10000 missing packet(s), 1 data break(s)

Event 2020-10-15-12h17m19UTC
2020-10-15-12h17m21-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 5232/10000 missing packet(s), 1 data break(s)
2020-10-15-12h17m26-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286799463.506004450
2020-10-15-12h17m26-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 5060/10000 missing packet(s), 1 data break(s)

Event 2020-10-15-12h39m31UTC
2020-10-15-12h39m33-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 7812/10000 missing packet(s), 1 data break(s)
2020-10-15-12h39m38-UTC>INFO...-[TolmFrameBuilderSource::AfterPageLoading] new source BsX (v5) fully activated at (corrected) GPS time: 1286800795.251104510
2020-10-15-12h39m38-UTC>WARNING-[TolmFrameBuilderSourceManager::MergeSubFrames] Source BsX: 2511/10000 missing packet(s), 1 data break(s)

 

Search Help
×

Warning

×