Virgo Logbook

Virgo Runs (O3)

berni - 22:58 Thursday 09 January 2020 (48108)

Operator Report - Afternoon shift

ITF locked for all the shift; Science mode was interrupted by an Adjusting break from 17:57 UTC to 18:02 UTC to tune the DIFFp_TY set point.

Sub-system reports

DAQ
red flag for DAQ_FFL__RAW_FULL_LATENCY, experts informed by mail.

EnvMoni
In the first part of the shift the ENV team did a sequential switch-off of the two WE UPS systems, see entry 48107. All the operations were done outside the building thus no need to break Science mode.

Images attached to this report

48108_1578606783_20200109222547_DAQ_FFL__RAW_FULL_LATENCY.png

48108_1578607118_Screenshot_2020-01-09 Plots for logbook report.png

Comments to this report:

masserot - 6:05 Friday 10 January 2020 (48109)

DAQ_FFL__RAW_FULL_LATENCY,

On resquest of N.Arnaud, L.Salconi setup a data transfer from the raw_full stream(stol03) to the /data/archive_rawfull at around 20200109-16h20LT .

The attached plots show for stol03 the increase of the network activity and the cpu usage related to this data transfer .

The consequence is a lost of frame files in the raw_full strean data due to the fact the FdWRawFull server is not able to write all the frame files:

2020-01-09-17h21m52-UTC>ERROR..-FdThread: frame 1262625680 lost; buffer full for thread TH_WR
2020-01-09-17h22m15-UTC>ERROR..-FdThread: frame 1262625700 lost; buffer full for thread TH_WR
2020-01-09-17h22m30-UTC>ERROR..-FdThread: frame 1262625720 lost; buffer full for thread TH_WR
2020-01-09-17h22m51-UTC>ERROR..-FdThread: frame 1262625740 lost; buffer full for thread TH_WR
2020-01-09-17h24m57-UTC>ERROR..-FdThread: frame 1262625860 lost; buffer full for thread TH_WR

Images attached to this comment

Non-image files attached to this comment

stol03_FdWRawFull.txt

mantovani - 8:23 Friday 10 January 2020 (48111)

I had to bring back to the old value the DIFFp ty setpoint (to -0.024) to make the SDB1 diffuse light structure to be reduced

salconi - 11:18 Monday 13 January 2020 (48136)

The frame loss has been solved on Friday by the IT Department setting up a volume, taken temporarily from the storage farm O3 rawdata pool, where the on-line rawfull stream from stol03 has been written during the weekend while the copying of the requested rawfull files was completing, so relieving the load on the on-line disk array used by stol03.

As shown by the attacched graph, the average "waiting for I/O" cpu time on stol03 has more than doubled since O3 start due to a number of additional DAQ fluxes to produce frames with various mixes of channels/latency/downsampling requested by commissioning.

This indicates that the current load on the on-line array has a limited margin for other DAQ fluxes to be added outside of the initial specifications without impacting the rawfull in-time DAQ production.

Images attached to this comment