Project

General

Profile

Running a novaddt process online

This page explains how to spin up a DAQ run on NDOS and FD such that it can receive DDT messages.

Quick Setup Instructions

1. Spin down the current DAQ run and bring down the DAQ system (including "Stop Run Control" and "Stop Resource Manager").

2. Log into the desired master node:

NDOS:
  ssh novadaq@novadaq-ctrl-master.fnal.gov

FD:
  ssh novadaq@novadaq-far-master.fnal.gov

3. FD Only: Change the underlying DAQ test release by moving the following configuration file into the config area (the first command makes a backup copy of what is currently in there, it might be that there is nothing in there in which case you don't have to worry about it):

cp ~/ddt/config/daq-operations.cfg ~/ddt/config/daq-operations.cfg.daq.original
cp ~/ddt/daq-operations.cfg ~/ddt/config/.

4. Restart the DAQ and bring it all the way through the "Prepare Configuration" stage (on NDOS, you may only have to stop the run to do this, but I have boot tested it yet). Then copy the Global Trigger configuration into place:

cp ~/ddt/GlobalTriggerRunConfig.xml ~/ddt/config/.

5. Take the DAQ through the remaining stages and start the run.

6a. You can now send trigger messages from the farm nodes:

NDOS:
  ssh novadaq@novadaq-ctrl-farm-##.fnal.gov

FD:
  ssh novadaq@novadaq-far-farm-##.fnal.gov

6b. Here is how you would start the trigger process on the farm node right out of the DDT base release:

setup_novaddt
source ~/ddt/ospl_setup.sh
ddt-filter -c NDOS-Trigger.fcl

7. After you are finished, switch back to the master node, bring the DAQ system all the way down again (including "Stop Run Control" and "Stop Resource Manager").

8. FD Only Wipe away the test release configuration file on the master node:

rm ~/ddt/config/daq-operations.cfg
mv ~/ddt/config/daq-operations.cfg.daq.original ~/ddt/config/daq-operations.cfg

9. The DAQ system is now in its original state, so you can hand it back to the shifter to start a regular run.

General Notes

  • Remember that the standard partition for NDOS is 0 and for FD is 1, so make sure your FHICL file sets the partition number as follows:
    NDOS:
    physics.analyzers.sender.rms_partition_number: 0
    
    FD:
    physics.analyzers.sender.rms_partition_number: 1
    
  • You can find the data here:
    NDOS:
    novadaq@novadaq-ctrl-master:/data/NDOS/
    
    FD:
    novadaq@novadaq-far-datadisk-03:/data-a/FarDet/
    

Development Notes

This note below include some working instructions for setting up a ddt process on NDOS. These instructions are a work in progress and will develop.

Initial setup

From a machine inside the FNAL.GOV domain connect to one of the NDOS buffer nodes:

ssh novadaq@novadaq-ctrl-farm-01.fnal.gov

Then setup the novaddt package, make and setup a test release:

setup_novaddt
mkdir tamsett
cd tamsett
newrel -t development DDT

You then need to copy Martin Frank's custom DDT packages. These differ from the trunk package in that they're setup to use the version of rms which is needed to run online. Once you've copied these go into include and setup links to them:

cp -r ~/mfrank/ddt/DDTDemo .
cd include
ln -s ../DDTDemo .
cd ..

Then link to his version of rms:

ln -s ~/mfrank/ddt/rms .

Then go into include again and setup the link for this:

cd include
ln -s ../rms/cxx/include ./rms
cd ..

You should check with the "ll" command that your include directory matches Martin's. Finally copy over the following script and source it:

cp ~/mfrank/ddt/ospl_setup.sh .
source ospl_setup.sh 

Now the DDT package is ready to go we can start a run on NDOS.

Package checkout

Once things are working in the head and you don't need to link against Martin's versions of things you can add in packages using CVS or SVN. For DAQ packages a command of the form:

addpkg -hd :pserver:anonymous@cdcvs.fnal.gov:/cvs/nova <package>

will work. Specifically for rms the command would be:

addpkg -hd :pserver:anonymous@cdcvs.fnal.gov:/cvs/nova rms
make rms.all

For DDT packages do a kinit <username> then:

addpkg_svn -h <package>

Spinning up a run on NDOS

End the run:

  • click end run in run control * click release resources * stop run control by clicking stop run control button (icon bottom left) * pop up appears and click partition-0, this is where run control is killed * check that resource manager shows null partitions (no partition 0), if it's there right click then release * stop resource manager (icon bottom left)

Now we're ready to add in our special DDT configuration, go to a terminal window and copy a config file that will overwrites which test release we'll use:

ssh novadaq@novadaq-ctrl-master
cd ddt
cp daq-operations.cfg_Partition0_stage config/daq-operations.cfg

Note this must be removed once we're done.

Start the run:

  • get a kerberos ticket (icon left) * start resource manager (icon bottom left) * start run control * select Partition-0 * click discover resources, select resources comes up, click ok * reserve resources, DAQ application manager appears - trigger scalars etc, wait for everything in application manager to go green * select configurations, select top from drop down * prepare configuration

Next edit/replace the DDT configuration xml file produced in the "prepare config" step previously so that it includes and enables the DDSNSet triggers and disables the beam spills and pulsers:

cp GlobalTriggerRunConfig.xml config/.

Finally load, make, configure etc, click all buttons all down until we're ready to begin run.

Sending the trigger signals:

From the buffer node we previously setup and run the filter:

ddt-filter -c job/NDOS-Trigger.fcl

And voila. Finally the data ends up:

/data/NDOS/*t12*

You can view memory from novacr02 and you can open the event view too (icon bottom right) (start auto advance).