Project

General

Profile

Dispatcher Instructions

Running a sample artdaq-demo system

Prerequisite

These steps assume that you have already installed and built the necessary packages using the steps listed on the Installing and building the demo page; as of Sep-19-2016 you will need to have installed artdaq-demo off of its develop branch for these instructions to apply.

Running a sample system

These instructions refer to the directory from where you "git clone"d artdaq-demo as BASE_DIR, using the naming convention of (Re)building the artdaq-demo package).

To run a sample system, consisting of 2 boardreaders, 2 event builders, and 2 aggregators, please use the following steps:
  1. in one shell window
    1. log into the test system where you installed the artdaq-demo
    2. switch to the bash shell, if needed
    3. 'source <BASE_DIR>/setupARTDAQDEMO'
      -> Henceforth in this wiki, the previous three steps will be referred to as "Set up the artdaq-demo environment"
    4. 'start2x2x2System.sh'
  2. in a second shell window, set up the artdaq-demo environment
    1. Make sure you're in the <BASE_DIR>/build_artdaq-demo directory, and then delete the FHiCL documents (files ending in *.fcl). They'll be created when you run init, below. If you run init without deleting them, the old *.fcl documents will be used, which may not be configured in the way assumed in this wiki.
    2. 'manage2x2x2System.sh init'
    3. 'manage2x2x2System.sh -N 101 start'
      • wait for a moment or two for event fragments to be generated, built into events, and written to disk (in /tmp). Please note that the output *.root file will get large quickly, so keep an eye on it while you run. The "-N" option is required, and allows you to set the run number (101 in this example).

The DAQ is now running. Once or twice a minute, in the window in which you launched start2x2x2System.sh, you'll see messages like

Mon Sep 19 13:40:33 -0500 2016: %MSG-i Aggregator:  Aggregator-mu2edaq01-5265 MF-online
Mon Sep 19 13:40:33 -0500 2016: Input statistics: 244 events received at 9.94941 events/sec, data rate = 0.0878255 MB/sec, monitor window = 24.5241 sec, min::max event size = 0.00882721::0.00882721 MB

or
Mon Sep 19 13:41:33 -0500 2016: %MSG-i Aggregator:  Aggregator-mu2edaq01-5265 MF-online
Mon Sep 19 13:41:33 -0500 2016: Received event 850 with sequence id 850 (run 101, subrun 1).

This provides us with basic metrics about the DAQ, but doesn't tell us anything about the contents of the data. For that, we'll want to perform monitoring. In a third shell window, set up the artdaq-demo environment, and run the following:
art -c ../artdaq-demo/tools/fcl/TransferInputShmem.fcl 

and what you'll see, twice a second, are messages like the following:
######################################################################

Run 101, subrun 1, event 95 has 1 fragment(s) of type TOY1

Toy fragment 0 has total ADC counts = 40

Fragment metadata: 
Board serial number = 999, sample bits = 12 -> max ADC value = 4096

First 10 ADC values in the fragment: 
...0:   1998  2282  2458  3187  1919  1381  2389  1993  1383  1864

######################################################################

Basically, one of the two boardreaders contains a fragment generator called "ToySimulator" which is creating fragments with a payload of 40 ADC counts apiece, where the ADC values are 12-bit and can therefore assume a maximum value of 4096. The values are randomly distributed according to a gaussian centered at 2048 with a sigma of 1/10th of the max value (i.e., 409.6). This monitoring you're running prints out the first 10 ADC values of this fragment from every 5th event passing through the system.

Let's play with this a little. First, hit "Ctrl-C"; you should see something like the following:

%MSG-i TransferWrapper:  ToyDump:toyDump@EndJob 19-Sep-2016 13:50:49 CDT  PostEndRun
Requesting that this monitor (shmem1) be unregistered from the dispatcher aggregator
%MSG
%MSG-i TransferWrapper:  ToyDump:toyDump@EndJob 19-Sep-2016 13:50:49 CDT  PostEndRun
Response from dispatcher is "Success" 

When you started art, a message was sent via XML-RPC to the second aggregator (the one not writing files to disk) indicating that you wanted it to send you events. The details (that every 5th event should be sent, and that it should be done via shared memory as opposed to, say, over a network via multicast) were provided in the FHiCL, along with a unique label for your process, "shmem1". The aggregator will refuse to accept your request for events if you don't provide it with a unique label - so if, e.g., you set up another artdaq-demo environment and tried to run the same script without having terminated the first instance, you'd get something like the following:
%MSG-i TransferWrapper:  art 19-Sep-2016 14:14:56 CDT JobSetup
Attempting to register this monitor ("shmem1") with the dispatcher aggregator
%MSG
%MSG-i TransferWrapper:  art 19-Sep-2016 14:14:56 CDT JobSetup
Response from dispatcher is "Attempt to register newly-created monitor with label "shmem1" failed; a monitor with that label already exists" 

However, since we just hit "Ctrl-C" on the first monitoring process, if the aggregator successfully unregistered it, we can fire it up again with the same FHiCL document (and hence the same unique label). However, before we fire it up again, let's change both the fraction of events we receive as well as the way we monitor them. Edit the TransferInputShmem.fcl file you provided the art process by changing its "modulus" variable from 5 to 100, and then change the following line:
  end_paths: [ a3 ]

to
  end_paths: [ a1 ]

Now run
art -c ../artdaq-demo/tools/fcl/TransferInputShmem.fcl 

once again.

What you'll see is two plots, which update about once every ten seconds, and which are filled with data from every 100th event. One shows all the ADC values for both fragments within a single event; notice that while the TOY1 fragment has 12-bit ADCs and hence a max ADC value of 4096, TOY2 has 14-bit ADCs and hence a max ADC value four times greater:

The other plot is two histograms, each showing the cumulative ADC values from all fragments of a given type. If you let it run for a while, you'll slowly see a gaussian forming.

When you want to stop the DAQ, from the terminal in which you've been running the "manage2x2x2System.sh" script, just do the following:

  1. 'manage2x2x2System.sh stop'
  2. 'manage2x2x2System.sh shutdown'
    • there will be a complaint in the first shell window that "stop" is an invalid transition; this is because as a failsafe to prevent shutdowns in the middle of data taking "stop" is automatically issued as part of "shutdown", but you just issued "stop" from the previous instruction. Don't worry about this.
  1. back in the first shell window
    1. type '<ctrl-c>' to stop the MPI program

Note that the output found in the terminal where start2x2x2System.sh runs is saved in the /tmp/pmt directory. More generally, diagnostic output can be found in the subdirectories of the directory defined by the "logroot" variable in the start2x2x2System.sh script.

artdaqDriver: a lightweight executable for testing

While the demo described above is designed to give a feel for what a full artdaq-based DAQ system looks like, frequently it's desirable to test fragment generators (discussed in-depth later in this wiki) and art modules in a more lightweight environment. Along with the standard BoardReaderMain, EventBuilderMain and AggregatorMain processes out of which one can create a DAQ, artdaq also supplies the artdaqDriver executable. Running artdaqDriver at the command line itself is very simple as all one needs to do is provide it with a FHiCL document; the complexity is in the FHiCL document itself. An example of this can be seen by the following steps:

  1. log into the test system where you installed the artdaq-demo
  2. switch to the bash shell, if needed
  3. 'source <BASE_DIR>/setupARTDAQDEMO'
  4. artdaqDriver -c fcl/driver.fcl

And you'll see output like the following:

######################################################################

Run 101, subrun 0, event 1 has 1 fragment(s) of type TOY2

Toy fragment 0 has total ADC counts = 100

Fragment metadata: 
Board serial number = 999, sample bits = 14 -> max ADC value = 16384

First 10 ADC values in the fragment: 

PRINCIPAL TYPE: Event
PROCESS NAME | MODULE LABEL.. | PRODUCT INSTANCE NAME | PRODUCT FRIENDLY TYPE | SIZE
Driver...... | TriggerResults | ..................... | art::TriggerResults.. | ...-
Driver...... | daq........... | TOY2................. | artdaq::Fragments.... | ...1

Total products (present, not present): 2 (2, 0).

######################################################################

Basically, in the driver.fcl document, it's specified that an instance of the ToySimulator fragment generator (the same type used in the 2x2x2 demo) generate and send a set of events, which are then processed by the ToyDump art module, found in artdaq-demo/ArtModules. You can open up driver.fcl and play around with things like the parameter settings for ToySimulator, the art modules used to process the events, or even the fragment generator used - on this final point, artdaqDriver is very useful for testing purposes when you're developing your own fragment generator.

EVERYTHING BELOW THIS LINE NEEDS TO BE UPDATED AS OF SEP-19-2016

Options

Many options for controlling the artdaq-demo system are done through editing FHiCL documents, as follows. After an edit, you'll want to rebuild the package for the changes to take effect.

Changing the type of fragments generated

Open up <BASE_DIR>/artdaq-demo/tools/manage2x2x2System.sh (or whichever script you're using to issue transition commands), and go to the line where the DemoControl.rb script is called. The two types of fragment which you can pass as arguments to DemoControl are "toy1" and "toy2", both generated by ToySimulator and having 12 and 14 bits per ADC value, respectively. The two used in the default manage2x2x2System.sh script in artdaq-demo are "toy1" and "toy2", but you can also try having both fragments be of type "toy1" or "toy2". Once you've edited the script, re-compile via the "buildtool" command (see Building the artdaq-demo package for more).

(Advanced) Changing the number of BoardReaderMain / EventBuilderMain processes

While the scripts shown above create a system with 2 BoardReaderMain, 2 EventBuilderMain, and 2 AggregatorMain processes, different numbers of these subcomponents can be used -- you can, for example, decide to use 3 BoardReaderMains instead of 2. In order to change the DAQ system's configuration in this manner, start by making copies of start2x2x2System.sh and manage2x2x2System.sh, giving them appropriate names (i.e., replacing the "2x2x2"); these scripts are located in <BASE_DIR>/artdaq-demo/tools . Then, do the following:

In the start*.sh script:

To add a boardreader, add a line that looks like

echo "BoardReaderMain `hostname` ${ARTDAQDEMO_BR_PORT[<N>]}" >> $tempFile

To add an eventbuilder, add
echo "EventBuilderMain `hostname` ${ARTDAQDEMO_EB_PORT[<N>]}" >> $tempFile

where you'll want to replace "<N>" with the appropriate integer -- so, for example, if there were already two BoardReaderMains and you want to add a third, replace "<N>" with "2" (note we're counting up from 0, not 1)

In the manage*.sh script:

Go to the line which calls the script DemoControl.rb ; using the same definition of "<N>" as above, if you want to add a BoardReaderMain, add the argument

 --<FRAGTYPE> `hostname`,${ARTDAQDEMO_BR_PORT[<N>]},<N> \

where <FRAGTYPE> is one of "toy1", "toy2", or "v1720"; to add an EventBuilderMain:
--eb `hostname`,${ARTDAQDEMO_EB_PORT[<N>]},$ebComp \

please note that the trailing backslash on every line is there because these arguments are meant to be part of a single logical line for the system to parse, even though in the file they'll be on physically different lines

Add the new scripts to the build environment

Once you've edited the scripts, they're not yet part of your artdaq-demo package. To add them in, you'll want to edit the <BASE_DIR>/artdaq-demo/tools/CMakeLists.txt file; all you need to do is put the two new filenames alongside the list of other scripts within the "cet_script(...)" parentheses at the top of CMakeLists.txt, and then run "buildtool" to compile them in (this assumes you've set up the artdaq-demo environment by sourcing the "setupARTDAQDEMO" file as described at the top of these instructions) . Once you've done this, you can use the new scripts as you'd use the start2x2x2System.sh and manage2x2x2System.sh scripts.

(Advanced) Changing the ports on which the artdaq processes run

When following the instructions for running an artdaq-demo DAQ session above, a default set of ports are assigned to the six artdaq processes running. If there's more than one user on a given system, it will only be possible for one of these users to employ the default set of ports. However, it's possible to override the defaults with a simple one-line change in both the start2x2x2System.sh and manage2x2x2System.sh scripts. Rather than the line

source `which setupDemoEnvironment.sh`

instead use the line
source `which setupDemoEnvironment.sh` -p <YOUR_BASE_PORT_NUMBER>

where <YOUR_BASE_PORT_NUMBER> typically defaults to 5200, but you should set it to another value (e.g., 5300). Once the edits have been made, perform a build of artdaq-demo so that the now-modified scripts will be copied to the build area from which they're actually run.