This afternoon we experienced problems on the Tango database cluster with one machine (olserve124) becoming unresponsive and the cluster doing automatic swapping on olserver125.
Unfortunately, in this situation the Tango database servers (DataBaseds) were anyhow not able to connect to the database and reporting the error:

Starting tango db server in itf domain as user uid=1228(virgod) gid=5000(cvs) groups=5000(cvs),1100(virgo) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
virgod@olserver129[~]: main(): arrived
Received a CORBA::Exception
Tango exception
Severity = ERROR
Error reason = DB_SQLError
Desc : Failed to query TANGO database (error=Got error 157 'Unknown error code' from NDBCLUSTER)
The query was: SELECT count(*) FROM property_device_hist
Origin : DataBase::check_history_tables()
Exiting

From the ITF/Automation point of view, processes like SatServer (interfacing with the Tango DSPServers) doing only parameters reading at the moment, were able to carry on, but as soon as the Automation (INJ_MAIN) tried to set a value for the LNFS, we got immediately the error:
Received set_8_ampl message from INJ_MAIN
2022-08-11 14h00m23 UTC
Error communication with LNFS
.......
desc = Failed to connect to device sys/database/2
....

so the locking procedure was stuck.

Continuing on the investigation of the problem, the machine olserve124 resulted corrupted (and need to be later restored) and to complete a working fail over on olserve125 the restart of the whole cluster was needed (which took some time).

Ultimately, as soon as the cluster was again operational, we could restart correctly the DataBaseds, resume standard operation for Tango and ITF locking could proceed.

This event (and few other recent occurrences of Tango infrastructure problems) is another push toward the removal of the very few remaining critical dependencies on Tango within the Automation/Locking procedure, basically involving only LNFS100, SatServer and the Virgopll SQZ servers. Failures on the remaining Tango process (providing maily data collection) sould not prevent locking.
Porting of Virgopll SQZ servers to pyserver is already ongoing and, since quite straightforward, we are also working on a new version of the PyLnfs100 server connectly directly to the LNFS box and making redundant the Lnfs100 Tango server.