Rogue DDT Processes¶

With the current DDTManager state model it can happen that some ddt-filter processes go rogue, meaning that they are not controlled by the DDTManager.

NDOS¶

1. Log into NDOS DAQ Master:

ssh novadaq@novadaq-ctrl-master.fnal.gov

2. Setup the DAQ environment:

setup_online

3. This command will tell you how many processes are running on each farm node (farm nodes 01, 02, 03, 05, 06, 07, 08, and 09 here):

rgang.py 'novadaq-ctrl-farm-{01-3,05-9}' 'ps x | grep ddt-filte[r] | wc -l'

4. Please think twice before killing, but here is how you kill all DDT processes on the same nodes as above:

rgang.py 'novadaq-ctrl-farm-{01-3,05-9}' 'killall ddt-filter'

FD¶

1. Log into FD DAQ Master:

ssh novadaq@novadaq-far-master.fnal.gov

2. Setup the DAQ environment:

setup_online

3. This command will tell you how many processes are running on each farm node:

rgang.py 'novadaq-far-farm-{01-40}' 'ps x | grep ddt-filte[r] | wc -l'

Note: you might be wondering the command showing above, here is an illustration:

[novadaq@novadaq-far-master /]$ ps x | grep DDTManager
17837 pts/6    S+     0:00 grep DDTManager
18159 ?        Sl     0:40 DDTManager -z 1
[novadaq@novadaq-far-master /]$ ps x | grep DDTManage[r]
18159 ?        Sl     0:41 DDTManager -z 1

4. When and how to kill DDT process
When: Only when shifters want to restart a run with a different configuration, or without any DDT process involved.

How:
You can simply stop the processes with DDTManager. If the DDTManager is not running, for instance: the shifters have released all resources.
You can do it on far master:

rgang.py 'novadaq-far-farm-{01-3,05-20}' 'killall ddt-filter'