Project

General

Profile

Running Code

  • VM access currently being implemented
  • setup_novaddt from Prelim Documentation
    • then from the command line you can try the test module
  • nova -c ddt-readFile.fcl -n 20 -s /nova/ana/trigger/data/ddt.root
    • The ddt.root contains a little over 5k events straight from the shared memory
  • Make a copy of ddt-readFile.fcl take out sample module, put in real module, presto

Can use files or arrange with DAQ group to get access to live data stream

Running on the Live Detector

When running on the live detector, care has to be taken to not conflict with the buffer node event build process that creates the data that we use. The way that this is handled is by "core locking" different running programs to different cores within the CPU. This is also referred to as "cpu affinity".

Taskset Method

The simple way of setting the cpu affinity for a program can be done using the taskset program and manually setting up where each process should run. This works well for starting specific process, but is not good for larger migrations of tasks between cores. See the next section (cpuset) for how to setup and use "sets" and masks for the cpu affinity.

The current method of doing this relies on the following assumptions:

  1. The buffer node event builder is running with two threads (standard config) and has been started and core locked to cores 0 and 1
  2. DDT processes are started with their cpu affinity set to exclude cores 0 and 1

Examples:
Starting a buffer node app on Cores 0 and 1

taskset 0x03 BufferNodeEVBapp ....

Starting DDT processes on cores other and 0 and 1 (assuming a total of 32 processor cores)
taskset 0xFFFFFFFC nova -c ddt-readFile.fcl

To check which cores a process (via its PID) is using:

taskset --cpu-list --pid <pid>

Cpuset Method

The cpuset method of setting the process affinity is a more robust system and is what should normally be used.

A tutorial on cpuset (cset) can be found at:
https://rt.wiki.kernel.org/index.php/Cpuset_management_utility/tutorial

Cpuset relies on setting up different areas of the machine into regions where different types of processes will run. And then creating "sets" of processes that run outside of these regions (i.e. you put all system processes on core 0 and then setup a set for ddt applications to have them run on cores 2-N). In the most basic "shield" model there are three sets: root, system and user. The "root" set is all cores and is what we think of as an unorganized processor. The "system" set is the area devoted to general system tasks that need to be run but are not time/resource critical. The "user" set runs in the "shielded" region of the processor and gets all the resources to itself.

First to use cset you must be root (or have cset setup to run suid). Then using this command:

cset shield -c 1-15

You setup a user region on cores 1-15 (inclusive) and a "shield" region for the system processes on cpu 0 (the compliment of 1-15). This will also migrate any system processes over to cpu 0.

If we want to also move all kernel threads (normally you don't want to do this) then you can do:

cset shield --kthread -c 1-15

To see the current status of the machine's regions:

[root@novadaq-far-farm-01 bin]# cset shield
cset: --> shielding system active with
cset: "system" cpuset of CPUSPEC(0) with 122 tasks running
cset: "user" cpuset of CPUSPEC(1-15) with 0 tasks running

For a more verbose description of where things are running:

[root@novadaq-far-farm-01 bin]# cset shield --unshield -v
cset: "system" cpuset of CPUSPEC(0) with 123 tasks running
   USER       PID  PPID SPPr TASK NAME
   -------- ----- ----- ---- ---------
   root         1     0 Soth /sbin/init 
   root       664     1 Soth /sbin/udevd -d 
   root      1543     1 Soth /sbin/rsyslogd -i /var/run/syslogd.pid -c 5 
   root      1544     1 Soth /sbin/rsyslogd -i /var/run/syslogd.pid -c 5 
   root      1545     1 Soth /sbin/rsyslogd -i /var/run/syslogd.pid -c 5 
   root      1546     1 Soth /sbin/rsyslogd -i /var/run/syslogd.pid -c 5 
   root      1555     1 Soth irqbalance 
   rpc       1569     1 Soth rpcbind 
   rpcuser   1587     1 Soth rpc.statd 
... 
   novadaq  31358     1 Soth BufferNodeEVBapp -z 1 --app-instance-name=bnevb...
   novadaq  31359     1 Soth BufferNodeEVBapp -z 1 --app-instance-name=bnevb...
   novadaq  31360     1 Soth BufferNodeEVBapp -z 1 --app-instance-name=bnevb...
   novadaq  31361     1 Soth BufferNodeEVBapp -z 1 --app-instance-name=bnevb...
   novadaq  31362     1 Soth BufferNodeEVBapp -z 1 --app-instance-name=bnevb...
cset: done

Here you can see that the buffernodeEVBapp software is has been migrated to CPU0. Since this is not what we want we can move it somewhere else:
[root@novadaq-far-farm-01 bin]# cset shield -s  --pid 31045 --threads
cset: --> shielding following pidspec: 31045 (with threads)
[==================================================]%
cset: done
[root@novadaq-far-farm-01 bin]# cset shield -s -v
cset: "user" cpuset of CPUSPEC(1-15) with 50 tasks running
   USER       PID  PPID SPPr TASK NAME
   -------- ----- ----- ---- ---------
   novadaq  31045     1 Soth BufferNodeEVBapp -z 1 --app-instance-name=bnevb...
   novadaq  31050     1 Soth BufferNodeEVBapp -z 1 --app-instance-name=bnevb...
   novadaq  31051     1 Soth BufferNodeEVBapp -z 1 --app-instance-name=bnevb...
   novadaq  31052     1 Soth BufferNodeEVBapp -z 1 --app-instance-name=bnevb...
   novadaq  31053     1 Soth BufferNodeEVBapp -z 1 --app-instance-name=bnevb...
...
   novadaq  31361     1 Soth BufferNodeEVBapp -z 1 --app-instance-name=bnevb...
   novadaq  31362     1 Soth BufferNodeEVBapp -z 1 --app-instance-name=bnevb...
cset: done

Here we have moved the buffer node event build (PID=31045) and all of its threads over to the non-system region.

If we want to start a program in the shielded region we can do so with the --exec option to cset. Here we start a bash shell in the shield area.

[root@novadaq-far-farm-01 bin]# cset shield --exec bash
cset: --> last message, executed args into cpuset "/user", new pid is: 19468
[root@novadaq-far-farm-01 bin]# cset shield -s -v
cset: "user" cpuset of CPUSPEC(1-15) with 52 tasks running
   USER       PID  PPID SPPr TASK NAME
   -------- ----- ----- ---- ---------
   root     19468 16788 Soth bash 
   root     20895 19468 Roth /usr/bin/python /usr/bin/cset shield -s -v 
   novadaq  31045     1 Soth BufferNodeEVBapp -z 1 --app-instance-name=bnevb...
   novadaq  31050     1 Soth BufferNodeEVBapp -z 1 --app-instance-name=bnevb...

Similarly we can start commands with user and group ids set properly:

root@novadaq-far-farm-01 ~]# cset shield --user=anorman --group=nova -e bash
cset: --> last message, executed args into cpuset "/user", new pid is: 1082
[anorman@novadaq-far-farm-01 ~]$ 

Note: to pass options to a program you need to separate the --exec progname from the arguments with a double minus "--" , so:

cset shield --user=anorman --group=nova --exec myprogram -- --progoption1=stuff --progoption2=morestuff

Here the shell was executed as me (anorman) instead of as root.

To tear down the shield model when we are done (and let everything migrate back across the cores:

cset shield --reset

What about the model where we want more than one shielded region -- i.e. a DAQ and a separate DDT region.

Here we define multiple user sets:

[root@novadaq-far-farm-01 ~]# cset set -c 0 -s system
cset: --> created cpuset "system" 
[root@novadaq-far-farm-01 ~]# cset set -c 1-2 -s daq
cset: --> created cpuset "daq" 
[root@novadaq-far-farm-01 ~]# cset set -c 3-15 -s ddt
cset: --> created cpuset "ddt" 
[root@novadaq-far-farm-01 ~]# cset set -l
cset: 
         Name       CPUs-X    MEMs-X Tasks Subs Path
 ------------ ---------- - ------- - ----- ---- ----------
         root       0-15 y     0-3 y   467    3 /
          ddt       3-15 n       0 n     0    0 /ddt
          daq        1-2 n       0 n     0    0 /daq
       system          0 n       0 n     0    0 /system

Now move all the system processes into the system set using the move syntax (the -f for from and -t for to):

[root@novadaq-far-farm-01 ~]# cset proc -m -f root -t system
cset: moving all tasks from root to /system
cset: moving 130 userspace tasks to /system
[==================================================]%
cset: done
[root@novadaq-far-farm-01 ~]# cset set -l
cset: 
         Name       CPUs-X    MEMs-X Tasks Subs Path
 ------------ ---------- - ------- - ----- ---- ----------
         root       0-15 y     0-3 y   333    3 /
          ddt       3-15 n       0 n     0    0 /ddt
          daq        1-2 n       0 n     0    0 /daq
       system          0 n       0 n   130    0 /system

Now move the buffernode event builder to the daq set:

[root@novadaq-far-farm-01 ~]# cset proc -m -p 31045 --threads -t daq
cset: moving following pidspec: 31307,31348,31349,31342,31112,31326,31346,31321,31344,31345,31098,31099,31360,31361,31362,31105,31107,31100,31101,31343,31045,31065,31064,31063,31062,31061,31060,31359,31358,31351,31350,31353,31352,31355,31354,31357,31347,31087,31066,31356,31088,31052,31053,31050,31051,31054,31055,31059,31076,31077
cset: moving 50 userspace tasks to /daq
[==================================================]%
cset: done
[root@novadaq-far-farm-01 ~]# cset proc -l -s daq
cset: "daq" cpuset of CPUSPEC(1-2) with 50 tasks running
 USER       PID  PPID SPPr TASK NAME
 -------- ----- ----- ---- ---------
 novadaq  31045     1 Soth BufferNodeEVBapp -z 1 --app-instance-name=bnevb00...
 novadaq  31050     1 Soth BufferNodeEVBapp -z 1 --app-instance-name=bnevb00...
 novadaq  31051     1 Soth BufferNodeEVBapp -z 1 --app-instance-name=bnevb00...

And now we can start our bash processes in the ddt set:

[root@novadaq-far-farm-01 ~]# cset proc -s ddt --user=anorman --group=nova --exec bash
cset: --> last message, executed args into cpuset "/ddt", new pid is: 28144
[anorman@novadaq-far-farm-01 ~]$ cset proc -l -s ddt
cset: "ddt" cpuset of CPUSPEC(3-15) with 2 tasks running
 USER       PID  PPID SPPr TASK NAME
 -------- ----- ----- ---- ---------
 anorman   6163 28144 Roth /usr/bin/python /usr/bin/cset proc -l -s ddt 
 anorman  28144 29454 Soth bash 

Now anything we start from our bash shell will automatically get locked into the ddt core set. So at this point we can fire up the ddt filters etc...

Scripts for automation

For convenince I've put together a few scripts to setup and tear down this infrastructure as well as migrate thing across sets:

setup_CpuCoreMap_daqddt.sh (sets up the right core tree)
reset_CpuCoreMap_daqddt.sh (resets all core trees to normal flat map)
migrate_CpuCoreMap_daqddt.sh (moves process to their proper sets)

Promoting Priorities

To get the most out of your DDT filter, you will need to adjust its priority. Since it will be locked to a core it is safe to let it have the highest priority since it won't interfere with anything else. To do this you need to renice the program AND all its threads (renice on just the parent won't work since the real work is done in subthreads).

This is done by getting the process id (PID) of the filter and then getting is group id.

pidof ddt-filter   <-- Gives process ids
ps -o pgid -p <pid>   <-- Gives group id of a process
renice -20 -g <gid>   <-- Bumps the priority of the process and all subthreads in the group

Put together:

ps --no-headers -o pgid -p `pidof ddt-filter` | xargs -I{} sudo renice -20 -g {}

Will bump all ddt-filters and their threads to max priority.

This is done in the startDDTFilters.sh script to start the filters.

FCL File notes

From ddt-readFile.fcl

Single file to switch between live stream and root file

Between Begin prolog and end prolog

Purpose of prolog
provide a bunch of stuff that can be plugged in anywhere in the fcl file

Things like timeout, stuff applied to source

Rootinput reader -> fileName: nil means you have to specify a name on the command line

services-> timing info, schedule in the end

physics block

@local anywhere in the file

add process.filter with your module
add a trigger path with your filter

possibly an output file with the events that survived the filter