Virgo Logbook

Virgo Runs (ER14)

magazzu - 23:03 Saturday 30 March 2019 (45476)

Operator Report - Afternoon Shift 30 March 2019

ITF found in LOW_NOISE_3_SQZ. Unlocked at 16:41 UTC, it wasn't initially possible to lock the OMC due to the Peltier disabled. I contacted the DET Oncall ( M. Eisenmann ) that started investigating the issue at LOW_NOISE_1, but at 17:37 UTC the ITF unlocked again and we started experiencing some strange behavior of the B7 signals ( see entry logbook #45474 ). This issue was solved at 19:33 UTC and it was possible to relock again, ITF back in Science Mode at 19:59 UTC.
ITF left locked.

VPM
ITF_LOCK, OMC_LOCK and DET_MAIN crashed after the unlock at 16:41 UTC. Properly restarted.

SQZ
ITF temporary exited LOW_NOISE_3_SQZ at 15:21 UTC, the Automation set it back to the same state a few seconds later. Back in Science mode at 15:22 UTC.

Downtime Event - Detection
ITF in TROUBLESHOOTING Mode from 17:07 UTC to 19:59 UTC. Initially, the mode was set due to the OMC peltier disabled after the unlock at 16:41 UTC, the period was extended because of the issue with the B7 signals.

Guard Tour
14:43 UTC - 15:19 UTC
16:49 UTC - 17:27 UTC
18:57 UTC - 19:21 UTC
21:10 UTC - 21:40 UTC

Comments to this report:

mwas - 10:05 Monday 01 April 2019 (45486)

At 16:42 UTC OMC_LOCK.py crashed while reading the parking temperature after the unlock (see below error message).

cm_send('SDB1_OMC', 'AcRelayChTranSet', 'OMC2_Peltier_stab_cmd_sw', 1)
2019-03-30T16:41:57.300Z OMC_LOCK W: Traceback (most recent call last):
File "/virgo/ext/metatron/metatron-1.4/lib/python2.7/site-packages/guardian/worker.py", line 461, in run
    retval = statefunc()
File "/virgoDev/Automation/userapps/OMC_LOCK.py", line 318, in main
File "/virgoDev/Automation/userapps/OMC_LOCK.py", line 86, in unlock
File "/virgo/ext/python/anaconda2/Linux-x86_64-CL7/lib/python2.7/ConfigParser.py", line 362, in getfloat
    return self._get(section, float, option)
File "/virgo/ext/python/anaconda2/Linux-x86_64-CL7/lib/python2.7/ConfigParser.py", line 356, in _get
    return conv(self.get(section, option))
File "/virgo/ext/python/anaconda2/Linux-x86_64-CL7/lib/python2.7/ConfigParser.py", line 623, in get
    return self._interpolate(section, option, value, d)
File "/virgo/ext/python/anaconda2/Linux-x86_64-CL7/lib/python2.7/ConfigParser.py", line 663, in _interpolate
    if value and "%(" in value:
TypeError: argument of type 'numpy.float64' is not iterable

As OMC_LOCK crash OMC1 didn't know it is not supposed to lock, so it kept trying to lock on the Michelson flashes. After a few minutes the security correctly turned on and put the OMC1 Peltier correction into disable.

OMC_LOCK has been restarted and then it must have read from file a very old parking temperature from OMC_LOCK.ini of 22.96 degrees at 16:44 UTC, and then 1 minute later read from the same file and find a more reasonable 23.26 degree set point. The OMC temperature set point of 22.96 is clearly wrong (300mK to low while it usually changes from lock to lock by 10-20mK), and hasn't been in the OMC_LOCK.ini file for at least the past 10 days. It needs to be understood why OMC_LOCK.py could have read some old version of OMC_LOCK.ini. This might also be a clue on why OMC_LOCK.py crashed in the first place (issue with OMC_LOCK.ini file reading).

swinkels - 11:29 Monday 01 April 2019 (45487)

The error can be reproduced like so:
In [1]: import numpy as np In [2]: import ConfigParser In [3]: config = ConfigParser.ConfigParser() In [4]: config.read("test.ini") In [5]: x = np.float64(1.23) In [6]: config.set('OMC1_T', 'temperature', x) In [7]: config.getfloat('OMC1_T', 'temperature') [... similar traceback ...] TypeError: argument of type 'numpy.float64' is not iterable
The problem is that configparser is only meant for storing and retrieving strings, and the user must do the conversion explicitly (with str and getfloat). This works:
In [8]: x = np.float64(4.56) In [9]: config.set('OMC1_T', 'temperature', str(x)) In [10]: config.getfloat('OMC1_T', 'temperature') Out[10]: 4.56
Why we don't see this issue in any other place where we write and read parameters is that the conversion to string is likely done implicitly when saving and reopening the ini file:
In [12]: x = np.float64(7.89) In [13]: config.set('OMC1_T', 'temperature', x) In [14]: with open('test.ini', 'w') as cfg: # save ini file ...: config.write(cfg) In [15]: config.getfloat('OMC1_T', 'temperature') # reading after a save gives the usual error [... usual traceback ...] TypeError: argument of type 'numpy.float64' is not iterable In [16]: config.read("test.ini") # reload In [17]: config.getfloat('OMC1_T', 'temperature') # now it works Out[17]: 7.89
So instead of doing a config.set(str(...)) everywhere, it might be enough to add a config.read(...) at the beginning of the unlock function.

swinkels - 12:12 Monday 01 April 2019 (45491)

I also had a quick look at the crashes of ITF_LOCK and DET_MAIN. Both are due to similar errors:

In /virgoLog/Automation/DET_MAIN/META_DET_MAIN_2019-03-26-20h28m15-UTC.log:2019-03-30T16:41:52.648Z DET_MAIN W: Traceback (most recent call last): [...] File "/virgoDev/Automation/userapps/DET_MAIN.py", line 132, in are_B1s2_PDs_enabled File "/virgoDev/Automation/userapps/META_library.py", line 290, in check_PD_enabled File "/virgo/ext/python/anaconda2/Linux-x86_64-CL7/lib/python2.7/ConfigParser.py", line 607, in get raise NoSectionError(section) NoSectionError: No section: 'B1_PD1'

And in /virgoLog/Automation/ITF_LOCK/META_ITF_LOCK_2019-03-30-16h43m50-UTC.log:2019-03-30T16:44:24.708Z ITF_LOCK W: Traceback (most recent call last): [...] File "/virgoDev/Automation/userapps/ITF_LOCK.py", line 482, in main File "/virgoDev/Automation/userapps/META_library.py", line 307, in ensure_PD_enabled File "/virgo/ext/python/anaconda2/Linux-x86_64-CL7/lib/python2.7/ConfigParser.py", line 607, in get raise NoSectionError(section) NoSectionError: No section: 'B2_PD2'

So it seems it cannot find the sections 'B1_PD1' and 'B2_PD2' in the file META_library.ini. This is weird, since the sections are there. Maybe some temporary glitch since the file is not readable??