In particular, he laid the pipes and filled the trench, a first layer of sand on the pipes and then the rest with normal earth. The trench dug 3 weeks ago.
This morning the TB compressor n2 went in alarm mode and stopped (picture n1), we have to understand the failure that probably caused a low oil level therefore we contacted the firm that took care of the maintenance that will be on site as soon as possible. However no problem for the ITF equipment that is under the compressor n1 as usual.
In meanwhile we noticed that the sensor used for monitor the Compressor n2 have a leak, therefore we dismounted and revisioned it on site, now it worked properly (picture n2).
This morning at about 6:30 UTC the burner of the WE heater stopped automatically due to the intervention of the electric thermal protection. After the check of the system the heater has been regulary restarted at about 7:00 UTC (picture attached), no variation in the temperature of the hall.
The fault has been fixed.
This morning the external firm performed the semestral maintenance of all the compressors on site, the operations have been completely transparent for the ITF thanks to the presence of double machines.
The fault has been fixed.
We always use a pair of water pumps for each water circuits (hot and cold water), one at a time. We have sent back to the company one of the two hot-water pumps of the WEB technical building (the one on the right) for routine maintenance, leaving only the left one running.
On Friday 11 Sept. we put the reiuvenated pump back into operation, taking a short break of the system for the time of the intervention (from 08:00 to 08:30 UTC).
We can see that the maintenance work had the effect of reducing the vibrations transmitted to the WEB seismometer (ENV_WEB_SEIS): the line at 47.2 Hz of the left pump was higher than the line at 47,1 Hz of the right pump.
Some sidebands have also disappeared. Paolo also noted that on the long term (one day) the 47.1 Hz line become even fainter.
Note that even when the pumps were noisy we had spring dampers under them, but their signal was still visible in the WEB seismometer. It seems there is a transmission path that is not coming directly from the pump to the ground, but maybe through pipes or water itself.
But if we perform periodic/scheduled maintenance, we can say that the signals of the pumps will not be visible in the seismic sensors.
This restarted when, having reached the site, I proceeded to exclude the secondary compressor, limiting the absorption on the switch.
Corrective actions to be implemented: check and correct secondary compressor failure; create two separate power lines for each compressor; optimize the alarm thresholds (on indications of the Vacuum group) to minimize downtime.
I have noticed that the ITF is not locking anymore. From a very quick glance at the DMS, the problem seems to be the temperature of the WE building out of control causing issues with the local control of the bench (to be confirmed by the experts).
Anyway I have tried to put the ITF in "locked arms" without success, while the North arm locks without troubles.
I have stopped the main Metatron node (ITF_LOCK) and put in auto mode CARM_DARM_LOCK with the status "locked north arm", hoping to keep a good reference for the alignment.
To be investigated tomorrow morning.
Such a temporary structure will be dismounted by December.
Meanwhile, it is not allowed to use it in that configuration.
As the external electrical company needed to switch-off the Central Hall lighting for at least one hour, it has been decided with Pasquale Popolizio to postpone the following works:
- positioning of an additional rod (15')
- positioning of two protected wooden slabs (15')
To be said that the slabs were not yet protected as the cleaning team was involved in the clean-rooms.
It is scheduled to begin the above mentioned remaining tasks tomorrow at 08h30.
The vacuum team will be then authorized to climb the structure (09h00)
Resetted Monday 17th at 8,45 a.m.
Normal level of pressure at 9,15 a.m.
Yellow flag for IMAC_LL_VF: percentage value of the valve of the cold water circuit of the cooling machine.
Red flag for IMAC_LL_ FL_VR: outlet air flow inside the cooling machine (alarm).
... but the investigation of the expert (D. Soldani) confirmed us that all the system is working properly... !
18871).
A comparison of the system response to the same kind of large disturbance (ie, a power cut-off of at most 90s duration), but with 3 different configurations of the controller, is reported.
- case A (in red): after the yesterday change (BP=6C, Tn=7.5‘);
- case B (in black: after the 5 Sept 07 modifications
(BP=8C, Tn=15‘);
- case C (in purple): before the 5 Sept 07 modifications (BP=40C, Tn=15‘);
The channel Em_TELS_ACS denotes the LasLab temperature, anthe standard value is 19.3C. The decrease of ~2C is due to
the reset of the controller consequent to the power cut (as explained in #18449.
The temperature of the LaserBench is on the channel Em_TELSLB02 and is the one to which the beam is more sensible and sholud change less than 0.1C.
The percentage of the electro-valve opening is on the channel IMAC_LL_VP_BP01.
In case A, the recovery time of the LasLab T is about half an hour (zoom on Fig.2) and such a transient is short enough to be
passively filtered by the acoustic/thermal enclousure of the bench and thus the transient is not seen on the T of the laser bench.
In cases B and C, the transient duration is longer and is seen on the T of the laser bench. Also, it can be noted that the controller needs some hour before asking again for the proper percentage of valve opening.
To be emphasized that for the events considered in cases B (#18437) and C (18617), the infrastructure on-call acted on the system by manually temporarly increasing the T set point,
otherwise the real recovery time would have been even longer. --> downtime of a few hours.
In case A, no manual interventation was performed. --> No downtime.
In conclusion, with the current controller configuratio, the Air Conditioning system is expected to react properly against power cut-off and therefore without affecting the good operation of the Injection System.
In detail, the changed parametrs are:
- the Proportional Band value was decreased from 8C to 6C;
- the integration time was decreased from 15‘ to 7.5‘.
Test #1
In order to check the effectiveness of connecting only the system controller and the sensors to UPS, a test reproducing such condition consisted of:
- ~8:15 UTC, power part of the plant (ie the sistem actuator) switched off for 90s (to reproduce an ENEL blackout).
Unfortunately, this action induced a reset of the controller, thus making useless this kind of solution. The reset of the controller is not understood, since itself and the sensors connected to it were left switched on.
The conclusion of this test is that it seems that the whole system should be connected to the UPS. Being the implementation of this solution more complex, its final evaluation is postponed after the run and the solution #1 proposed in #18449 was attempted in Test#2.
Test #2
Improvement of the control filter.
An optimal tuning of the filter parameters would require a full characterization of the system and therefore a quite long time. This is not possible during the run and therefore only a simple tuning was performed. It consisted in increasing the gain of the loop (the Proportional Band value was decreased from 40C to 8C). After this change, the system behaviour seems improved. Therefore we decided to leave the controller with this new value and keep it under observation during the next days.
In detail, the changed parametrs are:
- the Proportional Band value was decreased from 8C to 6C;
- the integration time was decreased from 15‘ to 7.5‘.
As usual, the blackout induces a reset of the controller of the LaserLab air conditioning system.
The controller is not enough reactive and therefore, it takes a long time (~ several hours) to integrate the error signal and to reach again the steady state.
Consequently, during this long transient, the correction computed by the controller is lower than the good one. A small correction means that the electro-valve that allows the flow of the warmed water is not enough opened. The warmed water is used in the post-heating unit to warm the cold air (@~17C) flow entering into the LaserLab.
Consequently, a small correction yields to a T lowering, as is now happening (2nd plot).
Fig2. is a zoom of the event.
A similar failure event occurred ( yesterday morning
In order to avoid such kind of failure, that usually makes longer the recovery of the ITF locking (~ hours + expert work),
a few upgrade actions (not nuch invasive) on the LaserLab air conditiong system could be:
1 - improve the tuning of the PI controller, to make it more reactive (this is a bit ctritical, since could induce system auto-oscillations);
2 - connect the controller and the drivers of the sensors to the UPS, possible also the electro-valve.
To be remarked that the connection of the remaininfg part of air conditiong system (ie the power part) to the UPS is not needed, considering that the blackout on the system can last at most 90 s (the time needed for the automatic engagement of the electrical generator) and that the absence of the actuation for 90 s doesn‘t produce significant changes on the T.
What is important to avoid is the reset of the controller filters.
Indeed, the LaserLab T (1st plot in the Figs), after a short increase, started to decrease (currently it is 18.9 C).
The effect on T of the LaserLab benches (2nd plot) is not yet visible.
The infrastructure on-call has been promptly contacted.
Currently D.Soldani and C.Fabozzi are working to restore the proper functioning of the air conditiong system.
The new data have been made available on the DAQ chain adding nitelServer as SMS provider within FbsMoni.
Here is the list and explanation of the newly available channels characterized by the prefix IMAC_LL (Infrastructure Monitoring Air Conditioning Laser Lab).
Take in account that, for consistency, the names of the channels has been kept the same as the one visible inside nitel even if not very well self explanatory.
The info listed here will be added into the Channed Database:
https://olserver13.virgo.infn.it/itf/channelsdb/
IMAC_LL_VP_BP01: percentage value of the post heating valve, (the one requested by the controller) in the laser lab
IMAC_LL_UR_RIPRESA: humidity of the inlet air of the cooling machine
IMAC_LL_VF: percentage value of the valve of the cold water circuit of the cooling machine
IMAC_LL_TA_1002: temperature measured by the sensor in the laser lab (the one used for the feedback of the ACS).
IMAC_LL_T_MAND: cold air temperature inside the cooling machine
IMAC_LL_ AL_FIL: differential pressure of the air filter inside the cooling machine (alarm)
IMAC_LL_ FL_VM: inlet air flow inside the cooling machine (alarm)
MAC_LL_ FL_VR: outlet air flow inside the cooling machine (alarm)
IMAC_LL_VP_BP04A: percentage value of the post heating A valve (the one requested by the controller) in the 1001 room;
IMAC_LL_ TA_1001: temperature measured by the sensor in 1001 room (the one used for the feedback of the ACS)
- at about 12:00 (LT) I have lowered the humidity set point to its nominal value of 50% ( In this way I have put the system back to the set point that was rised yesterday at 58%, after the intervention on the system at about 20:30 (LT)).
- at about 13:15 (LT) I have rised again the set point of the humidity at 58%.
NewTemperature set point = 22.5 C
Previous set point= 22.0 C.