Today it s the rtp15

2021-01-27-05h31m37-UTC>WARNING-FdFrMrgr:1295760707.0 frames from PAY_Img_50Hz start to be missing
2021-01-27-05h31m37-UTC>WARNING-FdFrMrgr:1295760707.0 frames from SQZ_Img start to be missing

Rebooted and servers restarted

The first part of the shift was dedicated to complete the activity on fiber coupling between EIB and Laser lab atrium; the activity concluded around 19:30 UTC.

After that started the planned ASC activit, activity concluded.

Other activities carried out:

- Installation in SDB1 Bench (still in progress at the end of this shift);

- SVS - Main and CC PLLs Commissioning (Vardaro, Zendri, Grimaldi)

- FLT - IN payload integration in clean room (Vocca, Travasso, Campeggi)

ITF left with cavities locked.

Air Conditioning
UTA INJ in "portata ridotta" at 19:30 UTC.

DAQ
at 15:30 UTC many Moni crashed , the processes were restarted without problems.

Once the signals in transmission were available, Alain has implemented the software changes needed to propagate the signals to the processes of ALS NEB. We re-tuned the triggers of the lock based on the new signals and we were able to engage the lock normally.

Today we have been working on the coupling of the green light from EIB towards the beating box in the laser lab atrium.

This morning, we installed the two fibers between the EIB and the box.

This afternoon we have been working on the coupling.

First we removed the photodiode looking at the beam coming from the west arm and installed the collimator at its place.
We used the green coming from the SHGB of the atrium to do a backward propagation on EIB and try to superpose the beam coming from the collimator and the ones coming from the interferometer.

The first approach in order to maximize the coupling into the fiber, was to scan the VCO in order to see an average power at the output of the fiber and maximizing this one. Unfortunately the power at the input of the fiber was only of about 100 uW which was too low to allow any coupling improvement.

The second approach was to keep only the beam coming from north arm and rotate the HWP in front of PBS separating the beam coming from north arm and west arm. By doing that we were able to have some power both towards the collimator and towards the photodiode. We relocked the injection and thanks to the power on the photodiode we were able to lock the green. We had about 3.2 mW at the input of the collimator. The green was sent towards the atrium in one fiber and bring back next to the EIB thanks to the second fiber in order to ease the lecture on the powermeter.
Both the injection and the green were unlocking which made the alignement complicated but we finally managed to get up to 1.4 mW. Considering the losses of the 30 m of fiber and the additional fiber connection, this is already a relatively good coupling. We decided to stop trying to improve it for now. 

We removed the second photodiode of EIB (the one looking at north arm) and roughly installed the collimator.

Henk Jan rebalanced the EIB from remote. Then we had to slightly realigned the green into the fiber.

Finally, we relocked the injection.

Conclusion: the green coming from north arm is injected into the fiber and is connected to the beating box.

Something very unexpected seems to have happened, which is most probably related to yesterday's activity, although the reason is still unclear. Anyway, for the time being, do not pause Metatron nodes, or they will eat up all the memory of olserver52 (this is probably the cause of the sudden death of many *Moni processes earlier this afternoon).

Verifications ongoing.

The support structure of the "cupola" and the "virola" of the FCIM microtower has been cleaned, assembled, transported and placed in the FCIM clean room.

The shift was dedicated to standard Maintenance. Here a list of the activities carried out:

• Standard Vacuum Maintenance (VAC Team);
• Cleaning inside Central Building, FCIM, and FCEM Clean Rooms (Menzione and Ciardelli with external firms);
• Optics Installation and fiber cabling in Laser Lab for ALS ( De Rossi, Gosselin, Montanari, Sposito );
•  Main and CC PLLs Squeezing Commissioning (Vardaro, Zendri, Grimaldi);
• IN payload integration in clean room (Vocca, Travasso, Campeggi );
• Installation in SDB1 Bench ( Gouaty, Spinicelli, Polini );
• Electric swap of the ACS INJ under UPS (D'Andrea with external firm, action performed from 7:10 UTC to 7:40 UTC)

The cavities remained locked for the first part of the morning, they were unlocked at around 9:00 UTC due to the beam blocked in Laser Lab. After the beam was available again at 12:30 UTC the flashes were still present on B7 and B8 and it was possible to relock without issue.
Activity in Laser Lab is still ongoing.

Air Conditioning
Under the request of the Commissioning crew, I switched the UTA INJ to "portata nominale" from 9:00 to 12:08 UTC. I set it back again at 13:45 UTC to its nominal flow configuration for the continuation of the activity in the Lab.

SBE
During the night the vertical correction of SPRB went over its threshold. From 7:23 UTC to 7:43 UTC I acted on the related motor3 to return the value around zero.

SUSP
(26-01-2021 12:55 - 26-01-2021 13:20) From remote
Status: Ended
Description: During the morning the current of the vertical coil of the PR F0 increased over their threshold.
Actions undertaken: I contacted the SUSP Oncall ( Ruggi ) that, using the relative motor, returned the coil to its nominal values.

Atter the rtpc6,  the rtpc7  , today the rtpc5 crahed .

After the reboot the servers were restartd successfully

Investigations inprogress

Observing the spectrum of RFC in loop error signal, one can notice that now it is a bit worse with respect to December (fig 1), when the loop was adapted to the new MC payload behaviour. In particular, the main difference is the height of the bump at the crossing of the UGF. In fig 2 a model of the open loop transfer function is shown, based on a measurement performed on December. One can see that the phase margin was a bit low (25 deg), and the bump a bit high (amplification of the noise by a factor of 2.5). A similar model, based on a recent measurement, is shown in fig 3: the phase margin is lower (20 deg) and the bump is higher (amplification 3.5). If one look at the situation as it comes out directly from the open loop transfer function measurements, this is even worse: the bump has moved from 3 to 4.5 (fig 4).

From the noise injections, one can extract the plant of the loop in the two cases, and try to understand where this lack of phase comes from. The old plant (fig 5) was well fitted adding a pole and a delay to the usual MC mirror simple pendulum. In the new plant (fig 6), the fit requires an increase of the delay from 560 usec to 660 usec. Then there is a new structure at high frequency, which is neither well measured nor well fitted, but is not decisive in the behaviour of the loop at the UGF.

Now it is worth to remind why a so critical loop was implemented for RFC lock a few years ago. The aim was to reduce as much as possible the line at 150 Hz in the frequency noise, because at a certain moment it became  huge and prevented a stable lock of the arms. Now the lock of the arms is again quite disturbed, and one possible reason could be the increase of frequency noise due to this bad behaviour of RFC loop. If this is the case, a possible improvement could be achieved in three ways:

1) adapting the RFC controller to the current plant and the current level of 150 Hz (never checked recently);

2) improving the plant, restoring at least the previous delay;

3) reducing the 150 Hz line at the source, if needed. It would allow the use a less critical RFC loop.

Few words regarding the 90 kHz bump visible in the TF measured as described in the entry 50391.

Its cause was a slight mismatch on the components of the LP filter present on the VCO board at the input of the receiver stage operating the differential to single-ended conversion for the DAC input used to tune the VCO frequency.

This has been addressed bypassing a couple of inductors on the board. After this modification the TF between the DAC differential input to the VCO tuning one as measured on the bench does not show anymore the bump. The same happens on the field, as reported in the entry 50482.

From the scan at 22.2 degres during polarization tuning one can try to measure the OMC finesse and sideband filtering.

Figure 1 shows the result. The time scale has been rescaled in MHz so that the 6MHz and 8MHz sidebands match their expected frequency (6.27MHz and 8.36MHz). Note that the scan is not perfectly linear, so to match the Airy peak to the carrier TEM00 (normalized to 1 at the peak), the lower and upper RF sideband are slightly offset. But their separation corresponds to the 2f (12.5MHz and 16.7MHz). To match the finesse of the cavity I assumed the OMC FSR is 834MHz (theoretical value), and increased the finesse until the tails of the Airy peak at ~20MHz started to match the data. This yields a finesse of 1100, higher than 900 measured at LAPP using full FSR scans. That higher finesse matches better the expectation from the coating specification that was 0.3% in transmission so a finesse of 1047 (to be checked against the measurement of the sample coated at the same time as the OMC).

The expected transmission of the 6MHz sideband from that Airy peak fit is ~3.7e-3. Slightly better than the 4.4e-3 transmission assumed when deriving the RAM specification VIR-1225B-19. So there shouldn't be issues with the 6MHz filtering.

Note that the P polarization peak lookgs high on this figure with 1% relative power. However the peak B1 PD3 power is 0.58mW, so this corresponds to a peak of 5.8uW, compared to the 58mW transmitted by the OMC. So it is actually only 0.01% of the total power.

Code in /users/mwas/OMC/OMC_scan_20210115

Today's activities focused on closing the ALS Green LASER drift control loop on the West Arm.
Furthermore, additional noise injections have been made in order to study the shape of the green LASER Open Loop transfer function at low frequencies on the North Arm.
 
The drift control implementation and optimization involved:

1. Porting the ACL code used for closing the drift control loop from the North Arm to the West Arm process;
2. Applying different values on the BPC loop tilt setpoint to study the drift control error signal gains  (See figure 1), and use it to close the loop.
3. Closing the drift control  loop (GPS 1295610150 )using the gains above (0.012 for TILT X, and 0.027 for TILT Y). This allowed to notice an oscillation @~0.03 Hz when the loop was enabled.
4. Testing different gains to improve the behaviour of the loop and suppress the oscillation. The tested gains were:
   • GPS 1295614697: 3X the initial gains. The oscillation of the loop increased a lot and caused multiple unlocks of the green beam.
   • GPS 1295615200: 0.5X the initial gains. The oscillations were reduced and this was the best gain (lowest RMS error signal, see figure 2) for the TILT X DOF.
   • GPS 1295617105: 0.25X the initial gains. This showed being the best configuration (lowest RMS error signal, see figure 3) for the TILT Y DOF.

In parallel, several noise lines have been injected on the North arm ALS LASER control loop as below:

ITF State: Arm Cavities Locked.
ITF Mode: Maintenance.
Quick Summary: IMC locked; all suspensions loops closed with the exception of SDB1; All SBEs loops closed; NARM_LOCK and WARM_LOCK in BOOST_ON; BS-SR Valve close.
Activities ongoing since this morning: Standard Maintenance, switch off of ACS INJ for Electric upgrade.

During this afternoon shift we attempted to put in operation the new metatron version as tested offline.
We soon discovered that the fixing of the latency was corresponding to the writing of metratron specific channel (like the node _index) with a second of additional delay (Fig.1) respect to the previous metatron version (See Fig.2)
After deep investigations, we resorted in spitting the frames interaction between check_connections() and the new manage_connections() ezca method to replicate the same situation in PyALP corresponding to the sequence:

• check_connections()
  • frame = fdio.get_frame()
  • out_frame = frame.copy()
• user code
• manage_connections()
  • out_frame.write_sms_prefix(.....)
  • fdio.put_frame(out_frame)
  • frame.close()

Many tests involving nodes stop/restart and NARM_LOCK + WARM_LOCK driving to locked were done from single user installation.
Then at the end of the shift a site installation of metatron 1.5 (containing the updated version 1.4 of guardian and ezca) was done and a last locking of the arms performed.
Fig 3 show the fixed latency of all restarted nodes and Fig. 4 show the absence of the additional one sec delay.

 

The main activity of the afternoon on automation went on for all the shift testing and debugging the new automation software, the work is still in progress...

Others activity carried out in the afternoon:

- DET: work inside the SDB1 tower;
- SQZ: activity inside the detection area (in progress);

Vacuum
(25-01-2021 18:15 - 25-01-2021 18:30) From remote
Status: Ended
Description: Close the BS-SR valve to allow the work inside the detection tower without laser beam.
Actions undertaken: Close the BS-SR valve.

Upon arriving on site I found that the West arm locked, while instead there were no flashes in the North cavity. Thanks to the recovery of the ISC Team both arms were locked again at 9:07 UTC.
The shift was dedicated to the planned ISC activity, carried out by Casanueva, Mantovani, Pinto, and Valentini. Also during the morning, the DET Tower was prepared for the planned Intervention.
The rest of the shift proceeded without issue to report, arm cavities left locked.

SUSP
At 23:35 UTC of Saturday, the ID of SDB1 opened. The loop was left open in preparation for the planned intervention in the DET Tower.

The unlock of this morning is actually due to a lost of lock of the RFC.
For an unknown reason the error signal became constant (see plot). 
Consequently the corrections ML_TH and ML_PZT increased till the unlock of the IMC. 

The IMC remained locked over the whole weekend.
It seems that, somehow, increasing the current of the ML pumping diodes, reduced the amplitude of the glitches on EOM_CORR (see plot 1 and 2). 
During the weekend the correction did not reach the +- 1V that makes the IMC unlock.

The only unlock was on the 25/01 at 06:12 UTC. It does not have the shape of the usual fast unlock. The correction of the ML_PZT starts to drift about 1 s before the unlock (see plot 3). This drift is not explain yet.

Last Wednesday (20/01/21) the bench SQB1 has been transported from the "locale camione" to the level of the DET lab with the support from EGO team (R. Rombolli).

Then on Thursday (21/01/21), the bench has been inserted inside the vacuum tank. We had some issue to insert it because of the cart being too high by a few millimeters, but finally we managed to insert the bench in the vacuum tank by removing the feet of the frame of the bench. Now the bench sits on the slides installed to insert the bench, we forgot some peek pieces (at LAPP) that goes underneath the feet of the bench, so they will be put back in place tomorrow once the peek pieces arrived from LAPP with LAPP workers.

The cabling of the bench was tested with the electronician team before integration in the vacuum tank, and the bench was recabled and tested again after integration. After correcting some faulty connection of one optical fiber, everything works as expected on the electronic side. All the eletroncis inside the electronic container is accessible.

We also realized that the lenght of the suspension wires is wrong, the 2 wires (one connected to the bench and the one connected to mSAS) are too far apart (108.4 mm). We will have to make a new cable to allow the connection and suspension of the bench.

Since the move to the new real time OS based on CentOS7, Linux 4.9.80 and Rtai 5.1, we observe jumps of 2 to 7µs in the TolmProcessor elapsed time, approximatively every 5 minutes. Looking at the logs, there is a match between these jumps and a disconnection from the ganglia server.

```
rtpc7 (SIB2_Tpro)

Jan 22 00:02:43 rtpc7.virgo.infn.it sshd[20579]: Connection closed by 90.147.138.128 port 57010 [preauth]
Jan 22 00:07:43 rtpc7.virgo.infn.it sshd[20598]: Connection closed by 90.147.138.128 port 59364 [preauth]
Jan 22 00:12:43 rtpc7.virgo.infn.it sshd[20628]: Connection closed by 90.147.138.128 port 33470 [preauth]
Jan 22 00:17:43 rtpc7.virgo.infn.it sshd[20647]: Connection closed by 90.147.138.128 port 35826 [preauth]
Jan 22 00:22:43 rtpc7.virgo.infn.it sshd[20677]: Connection closed by 90.147.138.128 port 38168 [preauth]
Jan 22 00:27:43 rtpc7.virgo.infn.it sshd[20696]: Connection closed by 90.147.138.128 port 40516 [preauth]
Jan 22 00:32:43 rtpc7.virgo.infn.it sshd[20725]: Connection closed by 90.147.138.128 port 42888 [preauth]

rtpc6 (SPRB_Tpro)

Jan 22 00:02:03 rtpc6.virgo.infn.it sshd[29051]: Connection closed by 90.147.138.128 port 59064 [preauth]
Jan 22 00:07:03 rtpc6.virgo.infn.it sshd[29070]: Connection closed by 90.147.138.128 port 33188 [preauth]
Jan 22 00:12:03 rtpc6.virgo.infn.it sshd[29099]: Connection closed by 90.147.138.128 port 35552 [preauth]
Jan 22 00:17:03 rtpc6.virgo.infn.it sshd[29119]: Connection closed by 90.147.138.128 port 37908 [preauth]
Jan 22 00:22:03 rtpc6.virgo.infn.it sshd[29148]: Connection closed by 90.147.138.128 port 40252 [preauth]
Jan 22 00:27:03 rtpc6.virgo.infn.it sshd[29167]: Connection closed by 90.147.138.128 port 42604 [preauth]
```

There is probably some disk/network activities initiated by the the ganglia process that impacts the interrupt real time latency. To be investigated.

The same issue occured again yesteday 20210124-21h58

In the attached plot one can see that when the issue occured on the rpc7 ;

• the slow frame builders without relation with some servers on the rpc7 have some troubles to collect theirs Slow monitoring servers
• the latency on the TolmFrameBuider increase during a short period of time
• Before the first issue one can see an smal increase and  some smal latency fluctuations on all the Tolm frame builder while before there was no fluctuations.These fluctuations are still here

ITF State: Only West Arm Locked.
ITF Mode: Commissioning.
Quick Summary: IMC locked; all suspensions loops closed with the exception of SDB1: its angular loop was manually opened for the planned intervention ( see report #50557), while instead, the ID opened during the night of the same day at 23:35 UTC; All SBEs loops closed; WARM_LOCK in BOOST_ON and West arm cavity, NARM_LOCK is attempting to lock but no flashes are present on B7.
Activities ongoing since this morning: no activity known so far in the control room.