Virgo Logbook

AdV-DAQ (Data collection)

masserot - 5:18 Friday 08 December 2023 (62676)

SDB2_Fb stopped(crashed)

The SDB2_Fb TolmFrameBuilder was found stopped (crashed) since 2023-12-08-02h59m14-UTC . The server has been restared at 2023-12-08-04h02m15-UTC.

Below the logfile's details of the FbmFFE Frame merger server collecting the SDB2_Fb frames

2023-12-08-02h59m14-UTC>WARNING-FdFrMrgr: Could not wait longer for frame parts, output 1386039563.8, nSources=15 first:SSFS_Fb
2023-12-08-02h59m14-UTC>WARNING-FdFrMrgr:1386039563.8 frames from SDB2_Fb start to be missing (isReady)
2023-12-08-02h59m14-UTC>WARNING-FdFrMrgr:1386039563.8 frames from SDB2_Fb start to be missing (out)
2023-12-08-02h59m14-UTC>INFO...-CfgReachState> Active(Active) Ok
2023-12-08-04h02m29-UTC>INFO...-Cm>ServerDoConnect> Connection in progress - CmName NULL, port 0, Host rtpc5.virgo.infn.it, owner virgorun, socket_id 11 (16384,425984), local 0, alive 1
2023-12-08-04h02m29-UTC>INFO...-FdFrMrgrFeed: 1386043366.8 receiving again frames from SDB2_Fb
2023-12-08-04h02m29-UTC>INFO...-FdFrMrgr:1386043366.8 frames from SDB2_Fb

Comments to this report:

masserot, mours - 15:58 Friday 08 December 2023 (62681)

The crash of this night’s TolmFrameBuilder SDB2_Fb is due to a lack of memory due to a memory leak (plot) present since April 2023 (plot).

After some investigations, the memory leak seems present on all the TolmFrameBuilders building a frame at 5Hz and filtering at 1Hz the pressure and temperature channels of the in air box photodiodes.

A new Fd's version will be delivered to fix this issue

Images attached to this comment

masserot, mours - 11:40 Sunday 10 December 2023 (62691)

The issue has been fixed in the Fd package v8r60p3 . A new release of the TolmFrameBuilder package v17r14p7 has been built and deployed on all the TolmFrameBuilder's server .

Operations performed between 2023-12-10-09h30m27-UTC and 2023-12-10-09h41m24-UTC

The attached plot (time as LT not UTC) show the memory usage on the SDB2_Fb's rtpc (rtpc5) according the Fd version: this TolmFrameBuilder was running the lastest release since 2023-12-09-11h54m13-UTC

Images attached to this comment