Rogue DDT Processes¶
With the current DDTManager state model it can happen that some ddt-filter
processes go rogue, meaning that they are not controlled by the DDTManager.
NDOS¶
1. Log into NDOS DAQ Master:
ssh novadaq@novadaq-ctrl-master.fnal.gov
2. Setup the DAQ environment:
setup_online
3. This command will tell you how many processes are running on each farm node (farm nodes 01, 02, 03, 05, 06, 07, 08, and 09 here):
rgang.py 'novadaq-ctrl-farm-{01-3,05-9}' 'ps x | grep ddt-filte[r] | wc -l'
4. Please think twice before killing, but here is how you kill all DDT processes on the same nodes as above:
rgang.py 'novadaq-ctrl-farm-{01-3,05-9}' 'killall ddt-filter'
FD¶
1. Log into FD DAQ Master:
ssh novadaq@novadaq-far-master.fnal.gov
2. Setup the DAQ environment:
setup_online
3. This command will tell you how many processes are running on each farm node:
rgang.py 'novadaq-far-farm-{01-40}' 'ps x | grep ddt-filte[r] | wc -l'
Note: you might be wondering the command showing above, here is an illustration:
[novadaq@novadaq-far-master /]$ ps x | grep DDTManager 17837 pts/6 S+ 0:00 grep DDTManager 18159 ? Sl 0:40 DDTManager -z 1 [novadaq@novadaq-far-master /]$ ps x | grep DDTManage[r] 18159 ? Sl 0:41 DDTManager -z 1
4. When and how to kill DDT process
When: Only when shifters want to restart a run with a different configuration, or without any DDT process involved.
How:
You can simply stop the processes with DDTManager. If the DDTManager is not running, for instance: the shifters have released all resources.
You can do it on far master:
rgang.py 'novadaq-far-farm-{01-3,05-20}' 'killall ddt-filter'