Other effects of the lightning: IT infrastructure

The lighting had effects also on other equipment in the computing room:

1. the ethernet interfaces on various servers (stfarm, ITF fileserver hypervisors, stol03) flapped at the time indicated
   (incidentally this caused the flapping of many iSCSI sessions on stol03 which triggered another pre-existing problem in the NFS mounting of some volumes (/data/trenddata, /data/rds,..)  causing the stucking of some proceses (MdVim ...) on olserver117, olserver117, olserver148 and  the need to reboot some ctrl workstations).
   No switches rebooted however.
   No effects on the electrical supply lines happened in this case.
2. an electrical line went in protection causing the down of the TCS cameras and of a switch connecting rtpc17 used to test the new CentOS7 rtpc baseline

It has to be noted that the only factor which distinguishes the servers impacted in case (1) respect to the other servers in the same room is that these servers are connected with direct attached, often long and poorly shielded, cables to their SAN switches and therefore were able to pick up enough inducted current from the lighting to trigger a misbehaviour of the connected ethernet ports. The majority of the other servers are instead connected directly to the switches integrated in the chassis with no cables.

If there is  the possibility that the presence of the magnetometers and environment monitoring cables passing from the roof station to the computing room has guided some of the lighting energy inside the room (as the current which broke the connected environmental monitoring electronics or their power supplies), either the location of this station should be changed, or some way should be found to separate galvanically the roof station from the inside equpment.

 

As the, hopefully, last effect of the lighting discharge affecting the computing room there was the  inconsistent state of a switch uplink which caused random NFS hangings on the volumes exported by 2 servers and consequently the difficulty to browse the /data directory itself.
Some VIM/Spectro building processes were therefore affected.

The volumes were:

/data/trenddata
/data/rds
/data/rawdata/rawfull
/data/archive_50Hz
/data/archive_trenddata
/data/seismon
/data/tcs
/data/squeezing
/data/GEOscience
/data/wdf

The reboot of the switch today at 9.35 LT has solved the problem transparently
 

Other effects of the lightning for the DAQ

100MHz Timing  distribution glitches

Due of the lightning some glitches have been seen on the Timing distribution for 100MHz Clock clock used in the CEB .(see the attached plots)

The re-configuration of the faulty  DBoxes allows to recover the correct running conditions:

• The Master and Monitoring DBoxes (CEB-DAQroom)
• the  LNFS and SSFS DBoxes (CEB-INJ_Elec_Lab)
• the SQZ_DBOX_2 (CEB-DET_EEROOM)
• the SPRB DBoxes (CEB -SPRB)

New Building

The IRIGB is not anymore available on the EGO-Optic-lab . We found the IRIGB available in the NewBuilding-network_room but it not anymore propagatd to the EGO-Optic-lab .

To allow the work of the EGO-Optic-lab the Timing signals are now generated by a DBox .

We found too that the channel 07 of DAC1955_SN58 is not working anymore

After some analysis and repairs, we can now be more precise in assessing the damage for these channels:

ENV_EXT_MAG_N
ENV_EXT_MAG_W

The "interface" box (with a battery inside, two switching power supplies and two balanced receivers/drivers) located on the grass between the magnetometers and the Central Building (NE side, on the roadside) has been repaired and checked; the battery charging system was broken. Unfortunately both magnetometers are dead. We have to plan to dig and recover them, send them to the company for repair, then bury them again.

ENV_ZeusAZ
ENV_ZeusCD
ENV_ZeusN
ENV_ZeusSpm
ENV_ZeusUD

This is the Boltek system (sensors on the roof over the Computing Room, receiving station inside the Computing Room), where the second lightning hit. Sensors has been melted (!), some thrown far towards the parking area. No hope to restore the system, we have to buy it again.

ENV_METEO_HU
ENV_METEO_PRES
ENV_METEO_RAIN
ENV_METEO_TE
ENV_METEO_WIND_DIR
ENV_METEO_WIND_SPD

This is the weather station (Davis Vantage Pro2) with sensors on a pole near the Boltek (a few meters) and the receiving/viewing station down in the Computing Room. It seems that only the receiving station is broken, we have tested the sensors with a spare station, battery powered, it works. All ancillary equipment (serial to RJ45, etc.) have not been tested. It should be easy to recover the signals by using the spare receiving station. No need to change the sensors on the roof (to be checked again with more experienced people on this system, e.g. Rosario De Rosa)

The external magnetometers were put back into operation in the late afternoon of Wednesday 17 November, 525 days after the damage caused by lightning.

The signals are really bad: over the time a new noise source has appeared. A comment on it will follow.

Today we compared the data of all experimental buildings magnetometers and the external one, looking at the first 2 harmonics of the strong narow-band noise that affects the sensors, namely at 5 Hz and 10 Hz. The analysis has been performed on today's data from 05:00 UTC with a 0.01 sec resolution. We found that the signal peak is stronger in ENV_CEB_MAG, followed in decreasing order by ENV_EXT_MAG_, ENV_WEB_MAB, ENV_NEB_MAG, whereas ENV_MCB_MAG data are unuseful because of the too high noise floor. Please note that the best Signal-to-Noise Ratios instead are reached by ENV_EXT_MAG, thanks to a lower noise floor. The complete data are reported in the table:

In the attached images there are the Amplitude spectral densities of the above sensors focused on the 5 - 10 Hz area.

The unknown magnetic noise source continue to exist. It has now some "hops" in frequency, and these seems correlated to the external temperature.

These lines are likely created by a PWM device, DC powered (otherwise they should be visible as 50Hz sidebands).

We see these lines in ALL the Virgo magnetometers, since 13 July 2021. But NOT in other environmental sensors, except for a vague correlation with the channel named ENV_CEB_ELECTRIC that sense the voltage drop between the vacuum metallilc chambers/tubes and the safety PE (Protection Earth) system.

A short survey with an external portable flux-gate magnetometer barely detect these lines, and their amplitude seems more or less the same in all the Virgo area.

We carefully checked the amplitude below the mains Medium Voltage aerial lines crossing Virgo, then above the Natural Gas pipe going parallel to the Virgo acces road. The magnetic signal is more or less always the same. No sudden and local increase.

We also checked nearest to the 3kV DC railroad, but the signal is no more present.

We finally found the source of the "Magnetic Monster" (as we friendly named it) that was plaguing Virgo external magnetometers, making them blind in the Schumann resonance band.

It is a 1V 2A DC power supply placed ... 4km from the CEB !

This is a system used to maintain an underground Natual Gas pipeline at a given voltage level relative to the ground, in order to avoid the so-called Galvanic corrosion. The pipeline starts from the SNAM Mortaiolo center station, towards Cenaia - Palaia.

This DC power supply, for unknown reason, modulate (chopper) its current a few times in one second (3 or 5 Hz). But it is not defective. Another good power supply put in its place have the same behavior. It is as if the resistance of the tube changes 5 times in one second (and this is impossible). Or, more likely, there is some other local current being injected into the tube and each power supply connected to it tries to counteract with another opposite-sign current.

Whatever the reason, we have been forced to leave things as they are for the moment. The SNAM company has become aware of the problem, they will call a meeting with the technicians to understand the reason for this behavior (not found in all the other pipellines starting from the Mortaiolo center station).

This hunt began in November 2021 when we discovered the M.M. and it took many people for many days to find the source. We measured something like a hundred places outside Virgo, with a portable battery powered system. The most active Noise Hunters were Jean-Loup Raymond and Lorenzo Pierini (thank you guys).

Attached is the map with a (presumed) drawing of the underground pipe in orange and the position of the DC power supply (the red square on the left), and the spectrogram showing the signal that disappears when we turn off the power supply.

Today at 14:31 UTC the SNAM staff set the galvanic power supply to "constant current" mode, removing the nightmare of the Magnetic Monster.

It still remains to understand the how and the why: we would like to avoid further problems in the future.