- carried out weekly checks for boiler, AHU and chillers in TB, CEB and MC
- refill of the LPG tank in Central area from 8.30 to 8.45 (UTC)
- carried out periodic check for AHU fan-motor belts state of tension and consunption (with external firm) with a short shut down of the involved machines:
AHU Clean rooms CEB: off at 6.40 - on 7.25 (UTC) ---> status OK
AHU INJ: off at 7.30 - on at 7.45 (UTC) ---> status OK
AHU DET: off at 8.53 - on at 9.08 (UTC) ---> status OK
- carried out weekly visual checks for boiler, AHU and chillers in TB and CEB
- refill of the LPG tank in Central area from 9.30 to 9.50 (UTC)
- carried out periodic check for AHU fan-motor belts state of tension and consunption (with external firm) with a short shut down of the involved machines:
AHU NEB: off at 6.47 - on 7.00 (UTC) ---> status OK
AHU WEB: off at 7.09 - on at 7.30 (UTC) ---> status OK
AHU CEB Hall: off at 7.50 - on at 8.05 (UTC) ---> status OK
AHU MC: off at 8.12-on at 8.26 ---> status OK
- Replacement of servo controls for the pre-heating and post-heating coils for the AHU DET Lab: task started at 6,30 (UTC); unfortunately, during this task, the 24 V transformer for the electrical power supply of the servocontrols inside the electrical cabinet was damaged; and it was necessary to replace it with a temporary one; this fault led to the restoration of operation of the air conditioning system at 5.30 pm (UTC), well beyond the maintenance limit.
Many Thanks to Massimo D'Andrea for providing the replacement quickly.
The new transformer will be available today and its installation, which requires a downtime of approximately 30 minutes, will be agreed based on the activity of the interferometer.
- carried out weekly visual checks for boiler, AHU in CEB, TB and MC;
- carried out weekly visual checks for boiler, AHU in NEB and WEB;
- refill of the LPG tank in Central area from 7,00 to 7,30 (UTC);
- repalcement of the broken air speed sensor for FC_EB clean room (AHU off from 8,50 to 9,00 UTC)
- carried out weekly visual checks for boiler, AHU in CEB, TB and MC;
- carried out weekly visual checks for boiler, AHU in NEB and WEB;
- refill of the LPG tank in Central area from 8,00 to 8,30 (UTC);
- installation of the dryer for air compressor n. 1(sx) in TB after maintenance in external mechanical workshop;
- check for 1500 West clean room AHU: replacement of servo controls for three-way valves necessary --> supply time 1 week
- carried out weekly visual checks for boiler, AHU in CEB, TB and MC with additional controls for drainage pumps in AHU CEB room (floating-switc regulation)
- carried out weekly visual checks for boiler, AHU in NEB and WEB
- refill of the LPG tank in Central area from 7,00 to 7,30 (UTC)
- carried out weekly visual checks for boiler, AHU in CEB, TB and MC
- refill of the LPG tank in Central area from 8,30 to 8,50 (UTC)
- carried out periodic maintenance work (with external firm) for steam producers for the AHUs of INJ, DET, FCIM and FCEM starting from 8,00 to 11,15 (UTC)
No activities in NEB and WEB
- carried out weekly visual checks for boiler, AHU in CEB, NEB, WEB and MC
- refill of the LPG tank in Central area from 10,15 to 10,30
- carried out periodic maintenance work (with external firm) for all the chillers on the site, including those dedicated to the systems for the experimental halls and laboratories; for the TB (for CEB and INJ and DET plants), NEB, WEB and MC buildings, since for redundancy reasons the chillers in question are all double, it was possible to carry out maintenance without interrupting the production of chilled water, by turning off one at a time.
However, the actual duration of the maintenance was from 8.30am to 1pm
- carried out weekly visual checks for boiler, AHU and chillers in TB and CEB
- refill of the LPG tank in Central area from 11,45 to 12,00
- carried out periodic check for AHU fan-motor belts state of tension and consunption (with external firm) with a short shut down of the involved machines:
AHU MC: off at 9.00-on at 9,15 ---> status OK
AHU CEB Hall: off at 9,21 - on at 9,28 ---> status OK
AHU CEB Clean rooms: off at 9,30 - on at 9,40 ---> status OK
AHU WEB: off at 9,46 - on at 9,55 ---> status OK
AHU NEB: off at 10,07 - on 10,15 ---> status OK
This morning the fancoil n1 get stuck and the temperature in the DAQ increased quickly (picture attached). I restarted successfully it at 7:55 LT and we informed the external firm in order to do further checks. Moreover I decreased the temperature of intervention of the third fancoil used as backup in order to prevent this kind of events.
these the time of shut down for each unit:
AHU Inj Lab: off 9,41 - on 10,12 (local time)
AHU Det Lab: off 10,25 - on 10,47 (local time)
All filters inside machine has been replaced; internal cleaning and sanitization; functionality control of internal shutters and their actuators; check correct seal between motor and fan
At WEB, also the cleaning of the Y-filter for post-heating coil was carried out; no time enough for same operation at NEB.
These the downtime per AHU (in local time):
CEB Hall AHU: off @8.33 - on @9.27
WEB Hall AHU: off @9.48 - on @11.15
NEB Hall AHU: off @ 11.45 - on @12.17
More precisely, the operating logic was not respected by the servocontrol which was positioned at maximum opening, causing the temperature in the laboratory to increase.
The effects of the replacement are yet to be verified in the operation of the conditioning system in the next few hours.
Unfortunately the work took longer than expected, as we had to face various critical issues related to the fuel supply and the internal cleaning of the machines, furthermore, the external company that was supporting us had to leave us to go to another emergency.
However, we finished the work ourselves, and managed to get everything back up and running.
on Friday 3 November at 09.00 am we started to have a series of interruptions to the ENEL electricity supply.
Our emergency network functioned regularly and the diesel generators were activated several times until around 11.00, the time in which we had a total interruption of the electricity supply due to a fault on the ENEL network, resolved at 18.45 about.
Our generators started regularly even on the last outage, but after a few minutes they went into forced stop by the engine safety systems.
We immediately intervened on the generators to understand what had happened, and we were able to verify that there was water in the service tank that powers the two 500 KVA generators that supply energy to the central building and the data center.
The gravity of the incident was immediately clear, and the impossibility of being able to do anything immediately, as the diesel engines had sucked water from the fuel circuit.
Aware that at that point only battery sources (UPS) remained to power our sensitive loads, I decided to immediately communicate to the control room that if ENEL had not restored the energy supply, in about three hours we would have been without energy. electricity in the central area.
Not being able to count on certain information from ENEL, the decision to turn off the electronic equipment in the central building and the data center was an inevitable action, in order to safeguard the equipment from an uncontrolled shutdown.
The UPS continued to supply energy until around 3.00 pm, after which they went into controlled shutdown to safeguard the residual life of the batteries.
We have made a new emergency connection on a new diesel tank to supply clean fuel to the generators, after attempting to purge the polluted fuel from the generators' power circuits, but the restart attempt failed.
The ENEL network returned to service at approximately 6.45pm, and we put the UPS back into service to allow the batteries to be recharged and to power all the security systems in the central area.
On Monday morning 6 November, further cleaning is planned on the diesel fuel supply circuits on board the generators, in the hope that there has been no serious damage.
If we manage to fully recover the generators, we will immediately return to ordinary conditions, with all sources of safety available.
However, if this further attempt also fails, we are already planning the installation of a backup auxiliary generator, which will allow us to have time to repair the broken machines.
So, in the best case scenario, we could return to normal operations on Monday late morning, while in the worst case scenario, we won't be ready until Tuesday evening.
Greetings
last night at around 7.30 pm, we had a failure in the mode cleaner's UPS.
I carried out an inspection to understand what had happened and I was able to verify that one of the two UPSs in parallel had suffered infiltrations of water coming from the roof of the building.
I tried to isolate the machine that I thought was damaged in order to restart the other one but unfortunately, there was a fault on the output board towards the load so both machines were affected by the fault.
I did not want to put the machines in by-pass on the ENEL network, as given the adverse weather conditions the network was not stable, therefore I would have exposed the sensitive equipment to continuous network spikes, so I preferred to postpone a more in-depth intervention until the following morning .
This morning, we were able to thoroughly inspect the faulty machine, and indeed we were able to detect the destruction of the machine's power board.
We isolated the faulty machine to be able to put the one we considered valid back into operation, and in fact the second machine suffered no apparent damage and started up again regularly.
The mode cleaner's UPS network returned to operation at 8.50 am powered by a single UPS, but this does not affect or limit the normal operation of the systems.
The repair of the faulty machine will take place as quickly as possible, we have already reported the request for intervention to the maintenance company.
Greetings
The two battery packs were installed in the truck room, as it was not possible to add the new pack near the existing one, which was installed near the entrance to the building.
With this new configuration, the machine doubles its energy autonomy in the event of a power failure, which goes from 20 minutes to 40 minutes at full load.
Having two separate battery packs also allows us to have greater safety in the event of a failure of one of the two battery packs.
The entire time we carried out the work, the UPS machine was positioned on "network 2" which is powered by the UPS that powers the network room, this allowed us not to expose ourselves to any risk coming from the ENEL network.
The machine was turned off this morning at approximately 9.00 am and the machine was put back into operation with the new configuration at approximately 4.30 pm.
Stress tests were carried out simulating various types of failure, the machine responded perfectly.
In order to reduce also NEB-AHU COLD loop oscillations, especialy associated with low temperature chiller regime, (according to WEB AHU PID Controller tuning), on September 13th, we modified the NEB-AHU PID Controller parameters, in detail:
14:35 UTC removed derivative term (from 90 sec to 0 sec) and increased proportional band from 3 to 4.
Fig.1 and Fig.2 summarizes consequences of our actions.
The removal of derivative term stops the oscillation (period 120sec). The residual oscillation in the correction signal, follows the chiller temperature (Fig2).
The reintroductiion of derivative term, 90 sec, on Sep 18th 8:30 UTC , in contrast with WEB, causes again loop instabilities (Fig3, Fig.4).
The derivative term has been removed on Sep 19th 13:58 UTC (Fig.5).
PID parameters:
Integration time: 360 sec
Derivative time 0 sec
Proportional band 4°
In order to reduce oscillations of WEB-AHU COLD loop, on September 4th, we modified the WEB-AHU PID Controller parameters, in detail:
9:11 UTC removed derivative term (from 90 sec to 0 sec)
11:45 UTC increased proportional band from 3 to 4
14:25 UTC increased derivative time fro 0 to 90 sec.
Fig.1 summarizes consequences of our actions.
In details, the removal of derivative term stops the oscillation (period 120sec). The residual oscillation in the correction signal, follows the chiller temperature (Fig2).
The reintroductiion of derivative term causes a further reduction of the temperature error signal (Fig1).
PID parameters:
Integration time: 360 sec
Derivative time 90 sec
Proportional band 4°
Temperature in the WEB hall dropped to 0,7 degree below the set point (22,5 C).
This morning at 8.20 (L.T.) I restored operation and the temperature of the cold lop returned to set (14 degrees) in 30 minutes.
At same time, to try to obtain a more rapid drop in room temperature, I have temporarily decreased the cold loop setting to 12 degrees.
At 9.40 (L.T.) the room temperature in the hall ("hot" loop) returned to its set point (21.5 degrees), and I gradually return the temperature of the cold loop to its previous set (14 degrees)
This is the list of the actions performed:
- thermal insulation of the room door (fig1)
- thermal insulation of the trapdoor (fig2)
- sealing of the passage holes for cables and pipes (figs 3-4-5)
- thermal insulation of the chiller pipe fittings (figs 6-7)
- moving the room temperature sensor to a position less subject to the fancoil air flow, and closer to the fancoil in-loop sensor (fig8). This is to have a better monitoring of the room temperature. The jump of about 1 degree in the temperature signal (fig9) is due to the new position of the sensor. No change in the set-point of the control loop.
Other actions may be done in the future, but to be agreed in advance with the TCS subsystem coordinator.
