To investigate this noise, the 

• The SBE_SWEB_LVDT_F0H {0,1,2}_raw  channels are acquired two times: the first time at 10kHz  for the loop and the second one at 50KHz for monitoring purposes. The 50KHz channels are stored with the "_FS" suffix
• The monitoring channels of the power supplies of the SWEB suspension bench are also acquired at 10KHz and 50KHz. The 50KHz channels are stored with the "_FS" suffix

These operations were performed

• at 2023-11-21-09h54m18-UTC for the SBE channels
• at 2023-11-21-08h45m05-UTC for the  SWEB_POWERSUPPLIES channles

After these operations the SBE loops were sucessfuly closed.

This plot and its zoom shows the SWEB_LC_LVDT_out channels acquired at 50KHz  for the following time period

• green: 2023-11-16-14h26m20-UTC
• red: 2023-11-16-18h26m20-UTC
• purple: 2023-11-17-00h26m20-UTC
• blue: 2023-11-17-04h26m20-UTC
• the color code is used for the following plots

From these plots, one can see that

• the peak is present on almost all the LVDT_OUT channels and moves from 7KHz to 13KHz during  this time period
• the peak is not present on the the LVDT_IN channels acquired inside the SWEB bench tank.

This plot shows the FFT-time of 2 SWEB LVDT_OUT sensors according the ITF state

• the 2 LVDT channels have the same noise form around 10KHz that drift in the same way over time.
• when the ITF  is locked at LOCKED_CARM_MC_IR state the noise seems stabilized around 11KHz .

This plot shows the same channels for  the SNEB benches:

• There is no noise on the SNEB LVDT_OUT channels
• the noise floor of the SNEB LVDT_OUT channels is lower  than 10e-5 V/sqrt(Hz} while the one of the SWEB channels is greater

These plots shows for the SWEB and SNEB benches

• the power supplies channels with a logarithm scale and a linear scale . Beware these channels are acquired at 10KHz only .
• the UPS and IPS monitoring channels (current and voltage)

Note : the LVDT box, the SWEB bench poser supplies and the DBox use to acquire all these signals are located on the same rack:  RACK E6

These plots , the plot and its zoom,  shows the ASD of  theLVDT_OUT channels, the SWEB POWER_SUPPLIES channels acquired at 50KHz and  ythe SEB_SWEB_LVDT_F0H_{0,1,2} ones for the following times

• green: 2023-11-21-10h-UTC
• red: 2023-11-21-11h-UTC
• purple: 2023-11-21-12h-UTC
• blue: 2023-11-21-13h-UTC
• According to these plots :
  • the peak is present on all the SWEB LVDT_OUT channels and on the SWEB power supplies channels too . These channels are located in the same rack (WEB-RACK6) and acquired by the same DBOX SN43
  • but it s not present of the SBE_SWEB_LVDT_F0H{0,1,2} channels acquired in the WEB-RACK5

This plot and its zoom shows the ASD and the coherence of the channels acquired in the WEB_RACK6 at 2023-11-21-13h-UTC

• there is a huge coherence between all these channels at the peak frequency, close to 1KHz at this time.

The last plots ( first (2023-11-21-04h_16h)and second (2023-11-21-10h_17h)) shows the FFT-TIME of  one of  the SWEB LVDT_OUT channels and the SWEB bench power supply channels for today at diffterent time periods

According to all these plots ,

• the problem seems related to the WEB-RACK6 rack
• a power-off/on of the DBOX has been already performed last January  without any improvments

As possible actions:

• check if there is still the cell-phones in the WEB-RACK6  and to move them outside using an other power outlet
• check if any ALS devices are connected on the same power outlet as the WEB-RACK6 devices
• others suggestions are welcome

Nicolas pointed out that the channel monitoring the quadrant current related to the p12V has a same similar shape as the line around 10KHz of the LVDT_OUT channel

• this plot shows for the 2021-11-21-10h-UTC  to the  2021-11-21-22h-UTC period
  • the green rectangle refers to the SWEB quadant current related to the p12V 
  • the red rectangle refers to a decrease of this current even if there is no modification of the DC mean values .
• this plot, related to the same previous time period, shows the time relationship between the noise of the WEB Bench power supply channel and the  p12V Quadrant current monitoring channel

The same plots related the 2023-11-16-10h-UTC to 2021-11-17-00h-UTC period :  trend - FFTTime

As  trial, at the next maintenance period it could be usefull to make some tests on power supply related to the SWEB quadrants .This power supply also provides the timing signal board : consequently the SNEB bench will need to be reconfigured

This morning,

• the SBE_SWEB_LVDT_F0H {0,1,2}_raw  channels monitoring readout at 50KHz  have been removed
• The monitoring of one LVDT_OUT (SWEB_LC_LVDT_BR_H_out_1MHz_FS) has been setip and now done at 1MHz

After these operations, Romain adjusted the positions of the SWEB bench to recover the ALS WEB lock

Yesterday, when some trials were made on the SWEB channels and server it appeared that the noise floor of the LC_LVDT channels changed, more precisely reduced.

The first attached file 2023-11-28-SWEB-operations.txt contains the details of all the operations performed on the SWEB LC setup.

The second plot is the FFT-time plot of one of the SWEB primary coils and its related second coil one. Beware these LVDT channels are the ones read by the SWEB_LC Acl task . One can observed that

• until the restart of the SWEB_LC server, the broadband noise in the 3KHz-19KHz frequency band is present  on the 2 coils
• and disappears or is reduced on the LVDT channels after the first restart of the SWEB_LC server at 2023-11-28-08h20m48-UTC
• this plot shows the SWEB LC pimary and secondary LVDT channels
  • for the LC and SBE loops closed : at 7h30-UTC and 9h01-UTC
  • for the LC and SBE loops  opened : at 7h32-UTC and 8h22-UTC

The RT Tasks repartition and scheduling for the SWEB_rtpc is the following

• RT_CPU0
  • Tpro: frequency 20KHz,  delay 35us
  • WEB_PCal_fast: frequency 20KHz , delay 35us, scheduling order 8
  • SWEB_Photodiodes: 10KHz,  35us, 7
• RT_CPU1
  • WEB_TIMING_moni: 10KHz, 35us, 6
  • WEB_NCal: 20KHz, 35us, 5
  • SWEB_PSD_sensing: 10KHz, 85us, 4
  • TCS_HWS_WE:: 100Hz, 85us, 3
• RT_CPU2
  • SWEB_SBE: 10KHz, 85us, 2
  • SWEB_LC: 10KHz, 35us, 1
  • SWEB_Quadrants:  10KHz, 35us, 0

This plot shows the trend of the RT_CPU2 task elapsed time : there is no major modifications after the SWEB_LC restart but it 's anayway strange that the restart of the  SWEB_LC server improve the shape of the LVDT signals read

Using the fact that the SWEB_Quadrants server is running the first one of the same RT_CPU as the LC and SBE Acl task , its elasped time has been increased from 2 to 4us

• 2023-11-28-10h28m10-UTC    info masserot     SWEB_Quadrants saved  masserot: Ser SWEB_Quadrants min elapsed_time to 4us (rev. 246777)
• 2023-11-28-10h28m10-UTC    info masserot     'Reload config' sent to SWEB_Quadrants
• 2023-11-28-10h28m44-UTC    info masserot     SWEB_Quadrants saved - masserot: Ser SWEB_Quadrants min elapsed_time to 0us (rev. 246778)
• 2023-11-28-10h28m45-UTC    info masserot     'Reload config' sent to SWEB_Quadrants

The same operation has been done on the SNEB rtpc too

• 2023-11-28-10h23m20-UTC    info masserot     SNEB_Quadrants saved - masserot: Ser SNEB_Quadrants min elapsed_time to 4us (rev. 246775)
• 2023-11-28-10h23m26-UTC    info masserot     'Reload config' sent to SNEB_Quadrants
• 2023-11-28-10h24m13-UTC    info masserot     SNEB_Quadrants saved - masserot: Ser SNEB_Quadrants min elapsed_time to 0us (rev. 246776)
• 2023-11-28-10h24m14-UTC    info masserot     'Reload config' sent to SNEB_Quadrants

This plot and its zoom shows the effect of the Quadrants readout elaspsed time increase:

• when the elasped time of the Quadrants server is increased , the noise flow of the  SWEB_LC LVDT increased too (plot)
• the same thing occurred for the SNEB LVDT too

For debugging, the LVDT signals provided by the ADC2378 mezzanines were made available only online with the prefix "LP_" thanks to a TolmFrameBuilder option . Using this prefix "LP_", these channels are available only online. This operation has been performed  of all the TolmFrameBuilders acquiring the LVDT channels

This plot shows that today these troubles are mainly present on the SNEB and can be present on the SWEB too. The LC and SBE Acl servers are running

• on the RT_CPU0 for SIB2, SPRB and SDB2
• while they are running on the RT_CPU2 for SNEB and SWEB

Some investigations were made to understand 

• the discrepancy between the LVDT channels sent by the ADC2378 mezzanines itself and theirs readouts by the rtpc
  • Moreover this is no message reported by the LC Acl server about channels delivered too late
• and the broadband noise in the 1KHz-20KHz frequency band:t
  • these channels are processed by the LC Acl task running on the RT_CPU2, but the RT scheduler is running on the RT_CPU0
  • for each cycle, after the completion of the DMA transfering the data from the Tolm board to the rtpc memory, the task ordering is well defined thank to  the channels dependency .
  • But it s not the case for the end the cycle,
    • When the ACL tasks executed on the RT_CPU0 are completed, the RT scheduler starts compiling the TOLM packets and sending them through the Tolm links and to the DAQ regardless of the status of the Acl jobs executed on the RT_CPU1 and RT_CPU2.
    • Only the temporal synchronization of the Acl tasks ensures the correct consistency of the data produced on these RT_CPUs, which means that the total elapsed time on each RT_CPU1 or RT_CPU2 must be less than that of the RT_CPU0.

Examining more closely the time elapsed by each of the tasks related to each RT_CPU  (NEB-rtpc8,  WEB-rtpc9) it appeared that for the odd cycles, the elapsed time of the RT_CPU0 is shorter thatn the ones for the RT_CPU1 or the RT_CPU2.

Using an Acl facility, one can increase the elapsed time of any Acl task . This was done the 2023-11-29 between 17h-17h30-UTC;

• This plot (zoom 6KHz-14KHz, zoom 9KHz-11KHz) shows the effect of increasing PCal Acl task execution time on RT_CPU0 for NEB and WEB rtpcs  .
• the bump between 500Hz and 20KHz  disappeared on the LC_LVDT channels acquired by the rtpcs

The last plots shows the FFT-time for the following days: 2023-11-29-12h_1day, 2023-11-30-12h_1day, 2023-01-12h_1day . From these last plots one can see that

• each time there a drift of the current related to the  +12V of quadrant, there is broadband noise in the 6KHz-14KHz frequency band.
• the drifts occurs independantly of the ITF's status, locked or not locked

So it remains to find the source of this noise and to check that the RT_CPU {1,2} time sequences are correct on each of the rtpcs