This issue occurs when the DaqBox + 12V power supply drops below the undervoltage trigger.
This is due to the voltage drop in the power cables inside the minitowers when there is too much current drawn, mostly when shutters are activated (open or close actions)

Both SDB2 DaqBox crashes (11/10 and 23/10) are due to a "B1p_PD2 shutter open" command while shutter B5_PD2 was closing. B5_PD2 shutter was triggered on a hardware currant limit (20mA)

In the attached figures, SDB2_POWERSUPPLY_DBOX_LEFT_DOWN_p12V is the voltage at the output of the power supply. There is voltage compensation at the minitower flange to compensate for the voltage drop in the cables between the power supply and this minitower flange. There is no compensation inside the minitower. An increase in SDB2_POWERSUPPLY_DBOX_LEFT_DOWN_p12V means that there is more current in cables and then a voltage drop at the power input of the DaqBox.

Looking at B1p_PD2 shutter activity, one can see that a lot of  "open B1p_PD2 shutter" commands are sent even if the shutter is already open. It would be better to check the shutter status before sending commands.

Today's shift focued on studying  and attempting to mitigate the interferometer working point drifts at carm offset zero in order to increase the lock stability.

Lock acquisition:
Both the CITF lock acquisition and the carm offset reduction up to carm offset zero behaved very reliably and allowed to perform a large number (>15) of attempts across the shift (see figure 1).

Stability investigation:
Serveral operations have been performed in order to increase the duration of the CARM offset zero state, which at the moment does not last more than a few (5/10) minutes:

• Normalizing the PR and DIFFp angular signals by the B4_12MHz_mag and B4_112MHz_mag signals respectively, in order to compensate for the gain loss of the error signals that is caused by the sidebands power decrease at carm offset zero (now automatically enabled when using the 'carm_null' functions).
• Tuning and engaging the COMMp vertical loop together with the horizontal one (now automatically enabled when using the 'carm_null' functions).
• Manually adjusting the BS (local control) working point in order to minimize B1p_DC, with some success in increasing the dark fringe stability.
• Tuning the dithering loops controlling the working point of the input and NE mirrors.
• Tuning the B1p_QD2_56MHz demodulation phases used by the DIFFp loop during the CARM offset zero state, since the optimal phase seems to drift in this state.

So far these attempts did not influence the sideband power loss rate and ultimately the unlock of the interferometer.

Further tests will be performed in the afternoon shift.

Today we worked again on the IR BPC.

First we put back the sensing matrix to identity matrix. Then we tuned the demodulation phase again. And checked each loop independently, they were working.

During the work of tuning the phase, we found the M3 X error signal in Q always had an offset, even the phase was well tuned. It seems we have some noise at 20Hz beside the line we injected. So we moved M3 X line to 45Hz.

Then we tried to closed all the loops together, the BPC could bring the IR transmission to about 11V.  We also tried to lock the cavity on IR, the BPC seems working better in this condition, because the IR transmission signal was cleaner in this case.

The time cavity locked on IR with BPC on was

09:53:00--09:55:30 (UTC)

10:05:03--10:10:44 (UTC)

The cavity unlocked due to the unlock of the RFC.

But then we checked the spectrum of the IR error signal, the lines we added seems contribute a lot to the rms(pic 1), especially the one at 25Hz(M3 Y) and 45Hz(M3 X).

So we tried to reduced the line amplitude and increase the Q of the shaping filter.  In this way, we also needed to reduce the gain of the control filter to avoid oscillation. 

The line amplitude for all the lines now are

IR_M3_BPC_X_line = 0.1

IR_M3_BPC_Y_line = 0.15

IR_M5_BPC_X_line = 0.15

IR_M5_BPC_Y_line = 0.1

The final phase for all DoFs are

IR_M3_BPC_X_phi0 = 1.78

IR_M3_BPC_Y_phi0 = 1.25

IR_M5_BPC_X_phi0 = 1.45

IR_M5_BPC_Y_phi0 = 1.60

After all this change, we compare the IR_error signal when we locked on green and IR. In pic 2, the purple line is when we locked on green and blue is when locked on IR. 

The lock on the IR was from 13:08:29 -- 13:12:10. The unlock was due to the unlock of the RFC.

The IR BPC can only be engaged when the IR transmission larger than 1V. It is better to engage the BPC with the AOM loop closed. Two new buttons have been added to the AFC process inside the ' IR BPC' section. When engaging the BPC, engage first the correction than switch on the BPC.

Today I created a new Python application hwsAna2021.py that supports all TCS systems: INJ, DET, NE, NI, WE and WI. With Ilaria we killed some ghost processes, we removed all old versions and we did first tests with the new version without finding major problems.

Upon arrival I found ITF locked on IR
SQB1_SBE loop opened and vertical position to restore. Operation performed at 06:20 UTC.
SQB2_SBE vertical position to restore. Operation performed at 06:30 UTC.

The planned ISC/TCS activity (carried out by Valentini, Mantovani, Ruggi) started at 07:00 UTC and went on without major problems for the whole shift.
During the lock acquisition, at ~07:10 UTC, WE and WI Guardians opened LC. Properly closed at 07:20 UTC. Tx and Ty positions restored and ITF relocked on IR.

Parallel activity
SQZ - Infra-Red controls of Filter Cavities, carried out by Yuefan, Polini

No Events Monitor to report

Additional details on the shift

• after the third hand off the vertical control of SQB1 bench opened. That's why I had to realign the IR with SQB1 M21 V
• I modified the offload logic: I multiplied the input signal with the lock flag and I modified the relay transition time from 100s to 1 s. In this way if the cavity unlocks the offload will open immediately without creating troubles.

Still to do:

• save as SMS channel the weight of the LFC_Z_INPUT in order to have in the data the transitions from green to IR
• Modify again the offload logic adding a second relay if we will discharge it again
• Modify the drift control trigger. When we lock on the IR the drift control works with the IR but its trigger is still on the green

At the end of the shift (about 24UTC) I tried to change the proportional gain of the Slow loop of the SC PLL (from 40000 to 30000).  It seems that it worked. We will monitor this in the following days

I have tried to check if the blending of the DC and Audio channels of the various photodiodes is up to date.

I have looked at time with the full interferometer locked (Oct 22 2021 12:40). Figure 1-5 shows the measured transfer function between the DC and the Blended channel. There is an issue on B4 PD2 and B5 PD2. For B4 PD1 things look fine (within the usual 5% amplitude error and 5 degree phase error). Also for B7 and B8 PD1 things looks fine as this has been retuned a few months ago.

For the photodiodes on SIB2 DC/Audio channel blending has never been implemented. B1p PD1 has it shutter closed, and the B1s and B1 photodiode have no beam due to the slow OMC shutter being always closed for the past few months. Also for B5 PD1 the blending has not been implemented.

Figure 6-7 using some arms locked data, B7 and B8 PD2 have a good blending filter (as expected as it has been updated recently).

ITF State: Cavities locked on infrared.
ITF Mode: Commissioning
Quick Summary: IMC and RFC locked; all Suspensions loops closed
- SQB1_SBE Position Loop opened and Vertical Pos to restore
- SQB2_SBE Vertical Pos to restore

All other SBEs loops closed.

Activities ongoing since this morning: No other activities

No Events Monitor to report

To understand the sensing noise floor for the HD AA loop I closed the SC shutter, I removed the HD_MIR to illuminate both HD PDs with the BAB reflected from FC (and with LO beam), I ran the HD_AA loop, then I stopped it and I measured the spectrum of the error signal without the dither lines (magenta trace in the first plot), with the usual value of the dither lines amplitude, i.e. 5. mV (orange), with 15 mV (green) and with 45 mV (blue). The SNR with 5 mV is less than 3 dB, and reaches a maximum with about 10-15 mV before saturating.

In the first plot the bandwidth of HD AA demodulation filters was increased from 10 Hz to 20 Hz. Afterwards I further increased it to 30 Hz and I engaged the AA loop. With gain -120000 a servo bump appears on the M4_X error signal around 8 Hz. I increased the gain of the other 3 d.o.f up to -180000 and a servo bump appeared on M6_Y as well, see second plot. With higher gain on M4_Y and M6_X the loop become unstable.

I restored the gain -120000 for all d.o.f. and I engaged the AA loop together with the CC loop, using a gain 150 for the fast CC loop. In such conditions the AA loop remained stable, however the M4_Y actuator saturated, probably to compensate the cross-coupling with the axial d.o.f. M4_Z.

I stopped the HD CC and AA loops, I restored HD_MIR, I opened the SC shutter, and I left the AOM loop for SC on FC engaged over night.

The reason why the SC AA was not working was because nonzero values were added to the error signals in the EQB1_HD_AA_dither.cfg file. Once corrected, the loop was engaged properly. We also updated the demodulation phase or SC_M5_X.

The aim of the today shift was try again to recover the stability of the IR lock of the filter cavity. 

Preparation of the system:

When I started I found a correction of a -45V in the GR_M7_Y mirror. Was a mistake left on friday at the end of the green matching shift. Then I tuned the green demodulation phase at 0.615 rad.

When I started to check the IR transmission from the filter cavity was about 1V thus I started checking the alignment.

1. I reduced the power of the SC and BAB by rotating the EQB1 FI waveplate by 15000 steps. Then I tried to engage the SC AA Loop. It did not work thus I aligned by end the SC. I did not debugged the loop. I think the problem arised on thursday morning when I restarted the HD AA process, Fiodor two weeks ago fixed this loop but maybe some parameter was not saved in the configuration file. We will check the loop the next week. In any case the SC was quite well aligned into the OPA
2. I started to align the cavity acting on SQB1 M21 and M22. At the end of the process I also checked the alignment sending ramps to I3 M3 and M5 mirrors. The transmission was about 10V whereas the BAB transmission was about 1.7V. I did not check the quality of the alignment searching the TEM10 mode
3. I checked the IR PDH signal and its amplitute was 1/3 of the usual one. Thus I checked the DC value of the EQB2_IR_PD_DC photodiode and I found -0.6V. I tried to align it with the EQB2_GALVO_5 but was already centerred, Thus looking into the camera I realized that the beam was not in the usual position. I moved SQB2_M23 picomotors and I recovered the power -1.6V and the usual amplitude of the PDH signal. I performed 700 steps in horizontal and 350 in vertical
4. I checked the SC PDH phase and was good but the sign wrt the green error signal was wrong thus I added pi rad to it. The new phase is 3.21 rad
5. I checked the green lock UGF and was 70 Hz thus I increased the gain from 1 to 1.3

IR hand off

After the alignment I removed the IR normalization:

• SC_PD_RF_NORM_SELECT was set to -1 i.e. forced to use the not normalized signal
• I performed the TF between LFC_GR_RF_5MHz_NORM and LFC_SC_RF_11MHz_FIL and I found a factor 8.6. Thus I modified the filtering weight into the configuration file dividing it by 8.6. It is changed from 0.034759 to 0,0040417
• I set the SC CND threshold to 5 for the rising and to 0.5 for the falling edge.

Then I started to perform some hand off. In total I performed 5 of them.

First Handoff 20:41 -> 20:47 UTC. I left the offload open (not checked why unlocked)

Second Handoff: 20:51 -> 20:57 UTC. Offload closed (lock trigger left on the green) (not checked why unlocked)

Third Handoff: 21:03 -> 21:13 UTC. Offload closed (lock trigger left on the green) It unlocked because I forgot to switch the error signal from SC to GR of the AOM loop but I swapped the sign. At a certain point the GR power dropped and it unlocked (figure 1). After this one I had to realigned in vertical the IR by 700 steps. Not understood why.

Fourth Handoff:  21:36 -> 21:39:36 UTC. Offload closed (lock trigger left on the IR) same mistake on the AOM loop but th trigger was on the IR, The unlock was caused by un unlock of the SC PLL

Fifth Handoff: 21:48 -> 22:23 UTC Here I solved all the mistake and I used the function for the handoff in the SQZ_init.py. I solved a couple of bugs in it. The lock lasted more than half an hour but it unlocked again for the SC PLL unlock.

It seems tthat now the IR lock is stable. We need to understand why the SC PLL started to unlock time to time. 

Dufing the shift the Green power dropped from 7.7V to 6.5V. I found that the beam moved on the left in the FCEB_GR_CAM. I restored the usual power by moving the set point of the drift control. In particular all the set points now are 0 except FCEM_DRIFT_TY_SET = 12. Now the power is again 7.7V. Also the IR power dropped from 9.8V to 9 V. Restoring green power I restored also IR power.

I left the cavity locked on green with the IR in resonance and the AOM loop closed.

Concerning the SQZ_init.py I tested only the last version of the IRhandoff() and the GRrestore() functions. The GRhandoff() will be tested soon.

In Figure 3 in violet the spectrum of the error signals when we were locked on green and in blue when we were locked on IR. 

During last maintanance we tested the effect of mechanical changes on two panels of the EQB1 acoustic enclosure. Using two uniaxial accelerometers attached to the border and the center of the panels, we performed tapping tests on:

a) one panel where some rivets were added to connect the two aluminum layers with the inner Pb layer (first picture);

b) another panel, besides the rivets an aluminum bar was added along the diagonal to increase rigidity (second picture);

c) a panel in the original configuration (third picture).

Between 8:33 UTC and 9:40 UTC we borrowed the cables of ENV_DT_ACC_Z and ENV_SQZ_PIPE_ACC_Y to acquire the signals of the two accelerometers on the DAQ. The noise spectra and corresponding TFs are shown in the attached plot, where green trace corresponds to modified panel a), bue trace corresponds to modified panel b), and brown trace corresponds to unmodified panel c). It seems that the extra aluminum bar is more effective than the simple rivets against the drum modes at few tens of Hz.

During the weekend I left the BAB alone on the HD cameras to observe the long term angular drifts. In 18 hours, until the FC unlocked, the beam position moved by less than 10% of beam radius horizontally and about 25% of beam radius vertically.

Profiting of no other activities on the interferometer, at the end of ISC/TCS shift, we tried to run the HWS-INJ acquisition with WI RH ON since the SLED beam reflected by the mirror is coming back on the  HWS sensor (fig.1 ). We experienced some software troubles probably related to the new matlab version (fig.2).

To be investigated  by experts.

This morning Maddalena changed the threshold of ALS North arm locking from 0.0022 a 0.0018 (alignment check should be done).

The CITF locked after the change of phase B2 56 MHz from 1.05--> 0.8 but it appears to be difficult to lock. Once reached this configuration, the ITF could not be locked at the CARM null.
The dark fringe appeared different wrt yesterday afternoon shift (fig.1-left yesterday evening, right this morning). The difference could be related to the thermalization of the CITF after the last TCS step performed yesterday few minutes before the acquisitIon of the CITF unlock. Thus it has been decided to come back to the Friday night TCS settings (see table below) to try to understand if it was easier to lock CITF and if it was possible to reach again the CARM null.

After the TCS settings change, Maddalena changed the following parameters for the CITF lock:

1) MICH  gain= 3.5 --> 2.5

2) SRCL gain=0.6 --> 0.4

3) PR TX gain=3.25 --> 4

The acquistion of the CITF lock became easier and the dark fringe shape appeared improved (fig.2) wrt the previous attempts of this morning and similar to the one acquired on Friday.
The ITF has been brought to CARM null in a sistematic way and the duration of the lock was always about 5 min.
From the TCS side, the target is to approach in different steps the same DAS power values of yesterday evening (see table below) checking all the figures of merit.

Thus:

1) At 10.20 UTC: NI CH power has been incresed up to 100 mW

2) At  12.03 UTC: NI OUT power has been increased up to 800 mW

3) At  12.32 UTC: NI OUT power has been increased up to 1.2 W

At any TCS step, Maddalena locked the ITF up to CARM null without issues, enganging with different timing the various control loops. 

Finally,  she enganged automatically DIFFp TX and TY,  COMMp TY in full bandwith, the PR translation and DIFFm and COMMm in drift control, at the end of CARM null step.

All the shift has been affected by strong wind gusts.   

The activity on ISC/TCS tuning proceeded without major issues, see dedicated entry for details.

From 10:00 UTC we experienced some troubles in lockngi the ITF due to the strong wind.

Activity concluded, ITF left locked on the IR.

SBE
SQB2 vertical position recoverd using step motors.

ITF State: cavities locked on infrared.
Quick Summary: all suspension loops closed; all SBEs loops closed (SQB2 vpos to be recovered); red flag for Fast_DAC - SQZ_CC_DAC.
ISC activity already started.

No DMS alerts during the night

We started with the same TCS actuator powers of yesterday evening (see entry 53617):

Michele reached CARM NULL around 07.55 UTC, but the lock lasted only for few minutes (see Fig. 1). Another lock at CARM null was achieved later on, with identical behavior of the ITF.

Thus, we  decided to get back to the TCS actuator settings of the 40 minutes long lock of October 11th (53456).

While the CITF was locked, the powers of the actuators have been modified to the following values:

During the thermal transient, the sidebands decreased to a new steady state value:

• 12 MHz: 0.115 → 0.08 mW

• 112 MHz: 0.65 → 0.50 mW

• B1p: 0.20 → 0.14 mW (probably due to the less SB content);

Fig. 2 and 3 show the B1p PC and camera images before and after the above DAS and CH tuning.

Once the thermal transient was completed (around 14.00 UTC), another attempt to reach CARM NULL was performed without success (53622).

We decided to recover the sidebands by tuning the CH powers with CITF locked.

@ 15:20 UTC, WI CH was increased from 100 mW to about 130 mW.

6 MHz SBs improved; 56 MHz SBs first improved, then started decreasing for no apparent reason (see Fig. 7).

Fig. 4 shows the B1p PC and camera images taken @ 15:30 UTC.

@ 15:34 UTC, NI CH was increased from 172 mW to circa 200 mW.

6 MHz kept improving, and the 56 MHz kept decreasing, showing no change or correlation with CH powers changes.

Fig. 5 shows the B1p PC and camera images taken @ 15:40 UTC.

The CITF unlocked and was relocked shortly after, Fig. 6 shows the B1p PC and camera images taken @ 16:03 UTC.

@ 16:08 UTC, WI CH was increased from 130 mW to circa 160 mW.

Fig. 7 shows the B1p PC and camera images taken @ 16:12 UTC.

Fig. 8 shows the trends of the signals during the TCS actuators tuning. While the quality of the B1p PC and camera images shows a clear improvement after the different CH tuning steps, the power in the 6 MHz sidebands did not reach again the initial value. The B4 PC and the HWS could have helped discriminate different working conditions. We decided to give a last try for CARM NULL, but the attempt died immediately before reaching the final state.

The aim of the shift was dedicated to checking the stability of the CARM null status after yesterday TCS tuning operation and in supporting further tuning activities (see dedicated entry).

At ~11.30 UTC we lost connection with two DAQboxes of SDB2 and  Pacaud intervened to restore it: 53619

At the beginning of the shift the ITF was locked and brought to CARM offset zero three times (with mild difficulties in achieving stable locks of the CITF) to check the stability of the current TCS configuration. One lock lasted only a few seconds, while the other two lasted ~8 minutes, and showed a quick degradation of the sidebands power that ultimate led to unlocks. Figure 1 shows a comparison between the first (continuous line) and the third  (dotted line) carm offset zero attempts.

The TCS configuration (Both DAS and CH powers) was therefore changed (see dedicated entry) at ~12.57 UTC while the CITF was locked. (See dedicated entry for details). The change resulted in a noticeable loss of sideband power (see figure 2) in the CITF configuration. Attempts of reducing the CARM offset after this tuning immediately failed (before the CARM to MC handoff), most likely due to the gain loss of the CITF DOFs, in particular when locked to the 3f. 

The CH was values were therfore changed at 15.20 UTC  (see figure 3) and allowed to partially recover the sidebands values. Since the 112 MHz was not recovered completely, the MICH and SRCL gain have been increased in DRMI_lock.ini from 2.5 to 3.5 and from 0.4 to 0.6 respectively in order to restore their UGF.

A last attempt to go to CARM offset zero (to evaluate the new DAS tuning) was performed at ~16.20 UTC but the ITF unlocked just before reaching the carm offset zero state.

The ITF was left with the arms locked on the IR beam.

The activity on ISC/TCS proceeded without major isssues; we had an interruption between 11:40 UTC and 12:10 UTC because two SDB2 DBOXES were no more reachable.

Activity concluded, ITF left in LOCKED_ARMS_IR.

DAQ
from 11:30 UTC red flag for Fast_DAC - SQZ_CC_DAC; experts informed by mail.

SBE
- SQB2 vertical position recovered using the step motors;
- SDB2 loops opened because of DBOXES problem; properly closed after having fixed the issue.

SUSP
WE and WI LC opened by the guardian at 8:10 UTC and 9:00 UTC; properly closed and mirrors realigned.

DAQ
(23-10-2021 11:45 - 23-10-2021 12:15) From remote
Status: Ended
Description: SDB2 DBOXES more reachable.
Actions undertaken: see entry 53169

Indeed with 20x less motion on SQB1 the position of reflected BAB on HD cameras is very stable, see attached plot.

Same issue today.

2021-10-23-11h59m25-UTC    info pacaud       Session start
2021-10-23-12h02m14-UTC    info pacaud       Disable power for device DboxLeftDown using PowerUnit09
2021-10-23-12h02m25-UTC    info pacaud       Enable power for device DboxLeftDown using PowerUnit09
2021-10-23-12h02m45-UTC    info pacaud       Reconfigure SDB2_DBOX_LeftDown in SDB2_dbox_bench
2021-10-23-12h03m19-UTC    info pacaud       Reconfigure SDB2_DBOX_LeftUp in SDB2_dbox_bench
2021-10-23-12h03m52-UTC    info pacaud       Reconfigure SDB2_DBOX_RightUp in SDB2_dbox_bench
2021-10-23-12h04m47-UTC    info pacaud       Reconfigure SDB2_DBOX_RightDown in SDB2_dbox_bench

The aim of this shift was to increase the Mode Matching  between the Green beam and the filter cavity. We worked only with EQB1_GLAOM and we reduced the Mode Mismatch to less than 8.0 %  (gps 1318946495 14:01:17 UTC). [Tem00 7.85; LG10 0.7 V]

We mounted the EQB1_GAOM  on a motorized slit in order to have a fine control of its position. After that we re-aligned the AOM in double pass configuration. 

We did not connect the slits to the DAQ system, but we tuned the position by hand, optimizing the resonance peak of the TEM00.  Before checking the high of the 2nd order peak we optimized the alignment using the Drift control and the BPC.

At the end off activities in DET lab, we lost the alignment of the Green beam: the power drop from ~7.6V to 4V. 

In order to recover the GR alignment we had to remove the set point of the Drift control, so the GR Beam now is lower on FCEB_GR_CAM (Figure 3). During the alignment activities SQB1 unlocked and it was recovered by the operator.

At the end of the Recovery we realigned the IR beam and we check the MM of the GR beam: we measured ~8% of Mode Mismatch (Figure 1 and Figure 2)

ITF State: cavities locked on infrared.
Quick Summary: all suspension loops closed; all SBEs loops closed;
No activity ongoing.

No DMS alerts during the night