Project

General

Profile

Configuring BufferNodeEVB memory buffer parameters

The BufferNodeEVBapp application is started from the DAQ system startup script "startBufferNodeEVB.sh" This, as of 2016.02.03, can
be found in /home/novadaq/DAQOperationsTools/bin, which is a symlink to some "Ops release".
The script specifies a few significant options (for example):

  --lo-water-mb=8700             - "mb" means mega bytes
  --hi-water-mb=9000
  --mblock-stale-tmo-ms=1800000

"mb" means megabytes.
"ms" means milliseconds.
The memory pool is limited by both time and memory size whichever is more restrictive.
The "lo-water" mark is used after the memory hits the "hi-water" mark at which point the application with
free pool memory down to the "lo-water" mark.

Shared Memory Segment

Introduction

The BufferNodeEVB and the DDT filter programs both use the ShmRdWr package. Currently the configuration parameters associated with the Shared
Memory are encoded into the value of an environment variable: SHMRW_KEY.
Both the BufferNodeEVB and the DDT filters must have this environment variable set and set to the same value in order for the data to be passed
between the 2 applications. The environment variable is set in the start up scripts:
startBufferNodeEVB.sh and
startDDTFilter.sh
Look for lines like:

export SHMRW_KEY=0x0001700F

Information on the encoding of the parameters can be found in the
ShmRdWr/cxx/include/ShmRdWr.h

These instructions will explain what a shared memory segment is, how the Buffer Node Event Builder can write to it, and how we can read out the information by means of a small test program.

In general, shared memory segments are sections in memory that different processes can access at the same time. You can use ipcs (Inter Process Communication Shell) to view the shared memory segments on your machine. For example, on novatest01 this might look like this:

[novadaq@novadaq-far-farm-83 ~]$ ipcs

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x0001700f 32769      novadaq    666        1895849984 14                      
0x5302000d 491522     novadaq    600        209715200  16                      

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x00000000 0          root       600        1         
0x00000000 32769      root       600        1         
0x0001700f 65538      novadaq    666        1         

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages    

We can then use the shared memory ID (shmid) to figure out what processes are communicating with a specific segment. For example, the first shared memory segment is currently owned by novadaq and has 14 processes attached (see nattch), its shared memory ID is 32769.

Quick Size Check

The size in the listing is 1895849984 bytes. The encoding of the SHMRW_KEY=0x0001700F is 16 buffers of 0x70+1 MB (or 113*1024*1024 bytes).
There is also one semaphore section that require a few kilobytes. So 1895849984 - 113*1024*1024*16 = 24 KB, which is close enough.

Listing Attached Processes

We can now use lsof (list open files) to see what the attached processes might be:

[novadaq@novadaq-far-farm-83 ~]$ lsof | grep 32769
BufferNod  3826   novadaq  DEL       REG                0,4               32769 /SYSV0001700f
ddt-filte 13226   novadaq  DEL       REG                0,4               32769 /SYSV0001700f
ddt-filte 13236   novadaq  DEL       REG                0,4               32769 /SYSV0001700f
ddt-filte 13246   novadaq  DEL       REG                0,4               32769 /SYSV0001700f
ddt-filte 13256   novadaq  DEL       REG                0,4               32769 /SYSV0001700f
ddt-filte 13266   novadaq  DEL       REG                0,4               32769 /SYSV0001700f
ddt-filte 13276   novadaq  DEL       REG                0,4               32769 /SYSV0001700f
ddt-filte 13286   novadaq  DEL       REG                0,4               32769 /SYSV0001700f
ddt-filte 13296   novadaq  DEL       REG                0,4               32769 /SYSV0001700f
ddt-filte 13306   novadaq  DEL       REG                0,4               32769 /SYSV0001700f
ddt-filte 13316   novadaq  DEL       REG                0,4               32769 /SYSV0001700f
ddt-filte 13326   novadaq  DEL       REG                0,4               32769 /SYSV0001700f
ddt-filte 13336   novadaq  DEL       REG                0,4               32769 /SYSV0001700f
ddt-filte 13350   novadaq  DEL       REG                0,4               32769 /SYSV0001700f

Displaying Detailed Shared Memory Information with ShmRdWrShow

Make sure the environment in which the ShmRdWrShow command is run has the appropriate
SHMRW_KEY environment variable. This can either be exported in the environment or specified
on the command line as in the following example:

[novadaq@novadaq-far-farm-137 ~]$ SHMRW_KEY=0x1700f ShmRdWrShow 
                     bufs_num=16
                    buf_sz_MB=113
                      grpsNum=1
                grpsSiz_pages=1
        bufs_largest_multiple=0xfffffff0
          largest_zero_offset=0x00000010
                 idx_wr_start=472989
           idxcnt_wr_complete=472989
                       offset=32
          total_bytes_written=0
                     semSetId=65538
                   writer_pid=0
idx  _bytes_  _1st_lword
  0  6064656  0x005c8a10 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
              0x00000000 0x00000000 0x005c89d0 0x0141a3ed 0x1b142a00 0x002bdd1b
  1  5935184  0x005a9050 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
              0x00000000 0x00000000 0x005a9010 0x0141a470 0x1d93d000 0x002bdd1b
  2  5991488  0x005b6c40 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
              0x00000000 0x00000000 0x005b6c00 0x0141a4f3 0x20137600 0x002bdd1b
  3  6049232  0x005c4dd0 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
              0x00000000 0x00000000 0x005c4d90 0x0141a576 0x22931c00 0x002bdd1b
  4  6128160  0x005d8220 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
              0x00000000 0x00000000 0x005d81e0 0x0141a5f9 0x2512c200 0x002bdd1b
  5  5995376  0x005b7b70 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
              0x00000000 0x00000000 0x005b7b30 0x0141a67c 0x27926800 0x002bdd1b
  6  6020672  0x005bde40 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
              0x00000000 0x00000000 0x005bde00 0x0141a6ff 0x2a120e00 0x002bdd1b
  7  5981856  0x005b46a0 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
              0x00000000 0x00000000 0x005b4660 0x0141a782 0x2c91b400 0x002bdd1b
  8  5971136  0x005b1cc0 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
              0x00000000 0x00000000 0x005b1c80 0x0141a805 0x2f115a00 0x002bdd1b
  9  6086160  0x005cde10 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
              0x00000000 0x00000000 0x005cddd0 0x0141a888 0x31910000 0x002bdd1b
 10  6170928  0x005e2930 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
              0x00000000 0x00000000 0x005e28f0 0x0141a90b 0x3410a600 0x002bdd1b
 11  6043824  0x005c38b0 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
              0x00000000 0x00000000 0x005c3870 0x0141a98e 0x36904c00 0x002bdd1b
 12  5931792  0x005a8310 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
              0x00000000 0x00000000 0x005a82d0 0x0141aa11 0x390ff200 0x002bdd1b
 13  6138240  0x005da980 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
              0x00000000 0x00000000 0x005da940 0x0141aa94 0x3b8f9800 0x002bdd1b
 14  6106032  0x005d2bb0 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
              0x00000000 0x00000000 0x005d2b70 0x0141a2e7 0x1614de00 0x002bdd1b
 15  6105280  0x005d28c0 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
              0x00000000 0x00000000 0x005d2880 0x0141a36a 0x18948400 0x002bdd1b
grpIdx=  0 idxcnt_rd_complete=472974
                 idx_rd_start=472974
                 total_missed=6825  6840
                       offset=0
                       semval=1

Now that we have a general introduction to shared memory segments, let us take a look at how we can write to them and read from them.

Writing to the Shared Memory Segment

In order to write to the shared memory segment on novatest01, the following two things need to happen:

  1. The Buffer Node Event Builder application needs to be running (BufferNodeEVBapp).
  2. We need to generate fake DCM (Data Concentrator Module) output that is sent to the Buffer Node Event Builder using an application called SimulatedMillisliceSender.

When both processes are running, the SimulatedMillisliceSender is sending millislices from a variable number of DCMs to the one BufferNodeEVBapp, which in turn fills the shared memory segment in a circular fashion. Currently, the millislices are all the same except for their time.

Buffer Node Event Builder (BNEVB)

To start up the BNEVB application, open a terminal and log into novatest01. After setting up the environment (first line) you can simply start the BNEVB application (second line) as follows:

source /nova/novadaq/setup/setup_novadaq_nt1.sh
BufferNodeEVBapp --app-instance-name='' --mfcfg='' --no-chksum

You can use BufferNodeEVBapp -? to see descriptions of all the options.

Pool Configuration

The BNEVB application maintains an internal set of parameters that allow you to configure the depth of its data buffers. These need to be configured correctly to allow the triggering system to properly locate and copy data segments out.

In particular the BNEVB maintains both a set of memory limits and a set of time limits on the data it keeps. These are represented as "high" and "low" water marks.

Memory Limits

The main memory limit that needs to be adjusted is the "low-watermark" which is the MINIMUM of amount of memory that the buffer node application will store. The value should be set in megabytes and represents that size of the pool that will be maintained. It is set from the commandline using --low-water-mb=<SIZE>

The "high-watermark" is the maximum amount of memory that the buffer node application will store. When it reaches this limit it will deallocate portions of its buffer until it reaches its low-watermark. The high-watermark is set by default to a value of 100% of the low watermark.

This value can be changed while the BNEVB is running by connecting to its configuration port which defaults to 7556.

(sleep 1; echo lo=<new_lowwater>)  | telnet localhost 7556

# Set the value to 1 GB
(sleep 1; echo lo=1000) | telnet localhost 7556

Time Limits

The time limits are used to determine what the age of the oldest block of data that is being stored can be. The time is set in milliseconds and is applied to the "milliblocks" that the BNEVB stores. If a milliblock is older than this value it dropped.

The time limit is set using --mblock-stale-tmo-ms=<time ms>

This value can also be changed while the buffernode application is running by connecting to its configuration port. The config port defaults to 7556 so you can connect via telnet:

(sleep 1; echo stale=<new_time>) | telnet localhost 7556

Example: set the time limit to 40 seconds

(sleep 1; echo stale=40000) | telnet localhost 7556

Fake DCM Output

In a second terminal on novatest01, set up the environment and start the SimulatedMillisliceSender with the following options:

source /nova/novadaq/setup/setup_novadaq_nt1.sh
simOpts="--pause-after-connect --pause-before-close --connections=50" 
SimulatedMillisliceSender $simOpts --no-chksum 10000

As with the event builder, you can see a summary of the available options using SimulatedMillisliceSender -?.

Reading from the Shared Memory Segment

Now that we are supplying the BNEVB with fake DCM data and are writing into the shared memory segment, we can use Ron's ShmMilliBlock package to read from that segment. Open up a third terminal, log into novatest01, go to your test release (or create a new one), set up the environment and check out Ron's package into the test releases:

source /nova/novadaq/setup/setup_novadaq_nt1.sh
srt_setup -a
addpkg -h ShmMilliBlock

Once this is done, we can compile the package together with its test program:

gmake
gmake tbin

Ron kindly supplied a test program with this package, so we can use it directly to read from the shared memory segment and to learn the syntax of his ShmMilliBlock class. The code can be found in ShmMilliBlock/cxx/test/ShmMilliBlockTester.cc and here is how you run the binary from your test release:

./bin/Linux2.6-GCC/ShmMilliBlockTester

Note that you can also simply type ShmMilliBlockTester, but the above shows the path of the binary.

BufferNodeEventBuilder Testing

Instruction and a snapshot of the results of some of the BufferNodeEVBapp testing is contained in the
document doc/testing_and_results.txt from the BufferNodeEVB package.

A link to this is contained here:
http://cdcvs0.fnal.gov/cgi-bin/fnal-only/cvsweb-oss.cgi/~checkout~/Online/pkgs/BufferNodeEVB/doc/testing_and_results.txt?rev=HEAD&content-type=text/plain&cvsroot=novacvs

wget -O testing_and_results.txt "http://cdcvs0.fnal.gov/cgi-bin/fnal-only/cvsweb-oss.cgi/~checkout~/Online/pkgs/BufferNodeEVB/doc/testing_and_results.txt?rev=HEAD&content-type=text/plain&cvsroot=novacvs"