From the LSC side , the data are transmitted in the following order to the Suspension DSPs  : NE, WE, BS, PR, SR, NI and WI  . 

As the additionnal delay is only for the WE and PR DSPs anf not for the BS  one : It seems difficult to justify that the delays observed come from the Timing part of the ISC rtpc

After Paolo has fixed a flag in the NI DSP, the calibration injections properly reach the DSP and are seen in Sc_NI_MIR_Z_CORR. The associated flag in the DMS indicate no problem in the CAL to Sc data exchange/delays, with the expected delay.

From the DMS this morning, it seems that not only WE MIR,MAR and PR MIR are not as we expect, but also NE MIR,MAR.

From the DMS values, we estimated the following offsets in the delay (compared to what we expect  and what we add before the WI shutdown):

• For NE, MIR,MAR,  ~25 µs.
• For WE, MAR ~75µs, for WE, MIR  ~50 µs..
• For PR, MIR, ~50 µs.

One should probably resynchronize the NE, WE and PR DSPs doing a re-synchronization of the DSP cards w.r.t. the timing system?

In order to further investigate the measured delay from CAl to Sc channel, we can look at the channels as TFMoni_NEMIR_CALtoSc_2012Hz_Phase,Coherence : they contain the results of the TF (coherence, modulus, phase) measured at the 2012.Hz line injected on all the CAL_*_MIR_Z_CORR and read back on all the Sc_*_MIR_Z_CORR.

As shown in figure 2 from this night, the coherence is very good when the suspensions are on local controls. The WI mirror received signal because the W arm was locked last night.

Figure 1 shows the coherence, modulus and phase for all the TFMoni_*MIR_CALtoSc_2012Hz_* channels. If we expect the communication/transfer function to behave as before the shutdown (and as during O3), the expected values are 0.94 in modulus and +0.487 rad in phase  (when the coherence is ~1).

This is the case for the WI, NI, SR and BS mirrors, but this is not the case for what concerns the phase for NE, WE and PR mirrors:

• NE phase is +0.805, i.e. -25 µs from the expected phase
• WE phase is -0.145 rad, i.e. +50 µs from the expected phase
• PR phase is -0.145 rad, i.e. +50 µs from the expected phase

Note that these values have been stable since last week.

The same checks can be done on the TFMoni MAR channels, but the expected phase value is around -0.145 rad.

-> such channels can be looked at when the suspensions are under local controls after the DSP is re-synchronized to validate it.

On Friday, around 13h34 UTC, Valerio has recynchronized the NE mirror DSP board (not the master board that was already re-synchronized the day after the electrical blackout).

This is clearly seen on the channel that monitors the CAL to Sc delay at 2012.5 Hz, as shown in the figure 1 (plot TFMoni_NEMIR_CALtoSc_2012Hz_phase) where the phase has jumped from 0.804 rad to 0.487 rad : it indeed came back at the expected level, the same as before the blackout.

Looking at the monitoring channels for the marionettes, figure 2, one can see that :

• The values for WI, NI and BS marionettes are as expected: modulus ~0.94 and phase ~-0.145 rad.
• The values for WE marionette are not as expected: modulus of 0.906 and phase of -0.778 rad, i.e. extra delay of -50 µs
• The values of NE marionette are not as expected: modulus of 0.906 and phase of +0.17 rad, i.e. extra delay of +25 µs

When the NE mirror DSP board has been resynchronized last Friday, it fixed the extra-delay of NE mirror CAL to Sc, but not the extra-delay of NE marionette CAL to Sc.

To conclude, probably the following boards must be resynchronized (assuming that the master boards are properly synchronized):

• NE marionette board
• WE mirror and marionette boards
• PR mirror board

Study of CAL to Sc delays after resynchronization of DSP boards on June 13

On Tuesday evening, Valerio has resynchronized the DSP boards: 13 June, around 14-17 h UTC. Some delays from CAL to Sc have moved to their expected values, but not all:

• figure 1 shows the TFMoni plots for the mirrors: all the mirrors have the correct delays after this intervention (phase 0.485 rad at 2012.5 Hz).
  • The WE and PR phases have moved to their expected value.
  • Note that from 15h30 to 16h40, the NE phase was not correct (-2.757 rad instead of 0.485 rad), corresponding to an additional delay of ~250 µs.  Was it related to any actions on NE DSP boards?
• figure 2 shows the TFMoni plots for the marionettes: there was no effect on the NE and WE marionettes, and their delays are not yet at the expected values (-0.1448 rad).
  • Why the resychronisation of the MAR DSPs did not work as the one done of the MIR DSPs?

Study of CAL to Sc delays during some tests in the communication from the RTPC to the DSPs

Alain made some tests yesterday morning, June 14, from 6h to 6h40 UTC (see logbook 60548) , delaying the time when the packets are sent from the RTPC to the suspension DSPs  by steps of 2 µs.

We expect that the delay from CAL to Sc channels increases by steps, either of multiple steps of 25 µs  EITHER of a single step of 100 µs (depending on how the DSP input packets are taken into account).

In this case, there is an effect (close to expected one) on the mirrors and marionettes with the correct delay from CAL to Sc, while there is no effect on the ones with a incorrect delay (MAR NE and MAR WE):

• figure 3 shows the TFMoni plots for the mirrors
  • all the mirrors have a correct delay from CAL to Sc (since June 13 afternoon), and the delays come back to the correct value at the end of the test by Alain.
  • all the mirrors show a phase jump during the scan of the communication delays by Alain. The phase jumps from 0.487 to -0.77 rad, which corresponds to a delay of ~90 µs.  To be investigated if the fact that we do not find exactly 100 µs as expected is due to some artifacts of the TFMoni plots (averaging?) or is real (and to be understood).
• figure 4 shows the TFMoni plots for the marionettes:
  • for the BS, NI and WI marionettes that have the expected delay at the beginning of the test
    • there is a phase jump during the scan by Alain. The phase jumps for BS and WI are from -0.1448 to -1.4085 rad, which also corresponds to a delay of 90 µs, as for the jump seen for the mirrors.  For NI, we see the start of a phase jump but the phase comes back to its correct value after 2 minutes, while TFMoni is doing averages over few minutes: hence we cannot really conclude on the amplitude of the jump, but the start of the phase decrease is similar to the start of the decreases seen on BS and WI (so the phase jump may be similar).
    • and the delays come back to the correct value at the end of the test by Alain.
  • for NE and WE marionettes, that do not have the expected delay at the beginning of the test
    • we do not see any phase jump during the scan by Alain. ?? This looks weird and must be investigated further.

Proposed further actions

• try to resynchronize again the NE and WE MAR DSP boards to check if it has an impact this time
• if not, resynchronize all the NE and WE  DSP boards together, including the master board?

• Also, in order to monitor the NI and WI communication, it will be useful that the DSPs sends out the channels Sc_NI,WI_MIR_LSC_CORR, as for the other towers.

The figure 1 shows the phase of the TFMoni mirror and marionette channels during the scan made by Alain yesterday.

CORRECTION to previous entry: Looking more closely at the phase jumps during the scan made by Alain yesterday, the jumps have an amplitude of 100 µs for NE, WE and BS mirrors (for which the duration after the jump is enough so that TFMoni (time averaged) channels converge towards a stable value after the jump).

ANOTHER VERY IMPORTANT CORRECTION:

• all the delays look fine, also for NE and WE marionettes!  See figure 4, after restart of TFMoni (12h50 UTC): all phases are at -0.145 for all the marionettes !  This is indeed confirmed by doing the TF from the raw data directly (see figure 5).
  • TFMoni has been restarted with less averaging for the CAltoSc channels in order to better monitor effects studied in the last days in the future.
  • We have to understand why TFMoni output was stucked for NE and WE marionettes!
• using the raw data to look back at the actions by Valerio on Tuesday evening, the delays from the marionettes were fixed by this action (see figure 6, purple is before the resynchronisation, black after, the phase has changed for WE MIR and MAR).

Figure 2 shows a zoom around the jump of NE and WE mirror phase. We see that the jump (100 µs in total) is made of 2 or 3 smaller jumps (smoothed by TFMoni averaging). They happen at the time when Alain increased by 2 µs the time to send the packets from the RTPC to the DSPs. And the time window of 3-4 minutes when they happen corresponds to the time window when Alain has seen lower coherence between the channel sent from the RTPC and the channel sent back by the NE,WE DSPs. This happens for scanned values in a range of 6 µs of the time to send Tolm packets.

In order to understand these behaviors, the explanation we have when everything is well synchronized is the following:

• the input TOLM packets are read and/or applied every 100 µs in the DSPs (not every 25 µs)
• the delay from CAL to Sc increases by 100 µs when the TOLM input packets arrival time passes from before to after the "10 kHz clock" of the DSP.
  • since the TOLM packet sending times have a jitter of ~6 µs on the RTPC, when they arrive close to the "10 kHz clock" of the DSP, there is some jitter in the communication, which explains both the loss of coherence seen by Alain between the input and output signals, and the delay increasing by steps when we are close to this condition.

This explains well the observations by Alain and by the CAltoSc measurements for the mirrors and the marionettes which are properly synchronized.

For the NE and WE marionettes which are not properly synchronized, a possible explanation is sketched in figure 3 (case of NE MAR DSP, with an  offset of 25 µs currently). The figure shows the clock ticks of  different components. The MIR DSP (in blue, which receives the TOLM packets), is in synch with the Master/DAQ DSP (in gray) and the TOLM packets (in red).  Probably the MAR DSP has its owns "10 kHz clock" (in green) delayed by 25 µs from the MIR DSP.

• in standard conditions, the TOLM packets at NE arrives a bit after the tick at 50 µs, so it is used by the MIR DSP at the next "big 100 µs tick" (blue point in MIR DSP line) and the data are sent back to the DAQ following the correct timing. For the MAR DSP, delayed by 25 µs, it is used at the next "big 100 µs tick" (green point in the MAR DSP line), and this delay is still seen when the data are sent back to the DAQ.
• yesterday Alain has delayed the arrival time of the TOLM packets in the window shown in red.
  • for the mirror data, as long as the TOLM packet arrives BEFORE the "big 100 µs tick" in blue, there is no effect. When the TOLM packets arrive AFTER, there is an extra 100 µs delay. When the TOLM packets arrive within 6 µs of the tick, there is some jitter in the communication (loss of coherence and extra-delay between 0 and 100 µs).
  • effect on NE and WE marioentte must be checked since at least TFMoni was not giving the correct values !! 
    • for the marionette data, since the scan did not reach the "big 100 µs tick" in green, there is no effect: this must explain why we did not see any extra 100 µs delay in the CALtoSc signals of NE marionette.

Yesterday June 16, around 12h50, broad-band noise injections were done to measure the response from CAL to Sc of all actuators. The following figures show the different measurements (blue) and the models inred (two poles fixed at 8174 Hz, and free gain and delay) with residuals.

The fitted gains are delays are the following (tower  level(MIR or MAR)    gain (V/V)  error_gain     delay (s)   error_delay:

GPS 1370954893
     NE MIR     0.999936  0.000000     0.000412 0.000000
     WE MIR     0.999936  0.000000     0.000412 0.000000
     NI MIR     0.995537  0.000000     0.000420 0.000000
     WI MIR     0.995260  0.000001     0.000421 0.000000
     BS MIR     0.999872  0.000000     0.000413 0.000000
     PR MIR     0.999801  0.000000     0.000420 0.000000
     SR MIR     0.999945  0.000000     0.000413 0.000000
 
GPS 1370955014
     NE MAR     0.995457  0.000000     0.000470 0.000000
     WE MAR     0.995457  0.000000     0.000470 0.000000
     NI MAR     0.995457  0.000000     0.000470 0.000000
     WI MAR     0.995457  0.000000     0.000470 0.000000
     BS MAR     0.994671  0.000002     0.000472 0.000000
 
All the delays from CAL_MIR,MAR_Z_CORR to Sc_MIR,MAR_Z_CORR channels are close to 412 µs for the mirrors and 470 µs for the marionettes (close to the value we were using during O3: 412 µs and 466 µs respectively)

It confirms that the DSPs are properly synchronized, as we found yesterday from the monitoring channels.