Project

General

Profile

Here are the basics for running the Run Control (RC) system in its entirety. The RC system has three main components.

  1. DAQApplicationManager
    This keeps tabs on all of the necessary processes running, including sebApp and assemblerMain instances, run control server, DDS connections, etc. You should use this GUI to start DDS on various machines, start sebApp instances, and start assemblerMain instances. Click around and get a feel for it. You can right click to start individual processes.
  2. NovaResourceManager
    I need to get around to changing the name to UBResourceManager or something... Anyway, this GUI will list the possible resources you are allowed to reserve in the "Null Partition" tab. If you open this and there is a second tab, "Partition 0", you likely want to right click it and "Release Partition". This is likely left over from the last time you used the system.
  3. NovaRunControl
    This GUI provides shiny buttons that you can press on to load fcl files, begin run, stop run, etc. For now, you should only use the expert buttons (click view-->expert) because the simple buttons aren't coded/debugged yet.

Here's how you run everything.
First, ssh into a machine as yourself (I recommend uboonedaq-evb for DAB cluster, ubdaq-prod-evb for LArTF cluster). Execute:

source /uboone/config/setup_RC_env.sh
. /uboone/config/start_RC_system.sh

The first will take a few seconds to execute. It sets up environment variables, and all UPS packages necessary to run the big-three listed above. The second will open the application manager and the resource manager, and will end with a cout to the screen telling you what to do next, including what is written next:

You have to have DDS running before you start run control, or assembler, or SEBs. You should first click on the DDS tab on the application manager and start DDS on all the machines you are using. You'll see that when multiple resources are set to run on the same machine, it is sufficient to start DDS for only one of them and the rest will turn green. You'll need an ospl.xml file in your ~ directory.

Once DDS is started, execute the script suggested by the output of the start_RC_system script: startRunControl.sh. The RC GUI should pop up.

Now, click around on the DAQApplicationManager GUI to start all of the resources you want started.

In Run Control GUI, start clicking on the RC GUI buttons, in right-to-left order from the top. Some notes:
  • In the "Select Resources" window, click on the resources you have already started in DAQApplicationManager. Beware, you should expand all drop-down menus and use the counterintuitive "disabled" box on the right to select ones you don't plan on using.
  • In the "Select Configuration" window, the GUI just does an "ls" of /uboone/config/RC_fhicl_files/ and parses it. If you want RC to pick up a fhicl file, you need to copy it here (and probably chmod 777 it).
  • Once you have selected the fhicl file you want, the "start dispatcher and online monitor" button will light up. While DAQAppMgr will show you if they are already running, you can only start them from run control, because they can't be started until you've chosen your fhicl file. If you aren't using these resources, you can just never click that button.
  • If you want a run number other than the default 0, type "set_run_number (number goes here)" in the "Execute command:" box before clicking Begin Run.

When you want to shut things down, you should probably do them in this order:
1) Use the DAQApplicationManager GUI to shut down your sebs, DDSes, assemblers.
2) . /uboone/config/stop_RC_system.sh (which is essentially just a bunch of fancy kill -9's, but will eventually include commands to gracefully shut down things once there's a way to do so).

Troubleshooting:

THERE WILL BE BUGS. REPORT THEM TO SOMEONE WORKING ON RC (for now, , in the future, more people).

Buttons not lighting up? If in doubt, use the following scripts to restart ResourceManager and RunControl: stopResourceManager.sh, startResourceManager.sh, stopRunControl.sh, startRunControl.sh (which are prepended in your $PATH, so you can just type these names in the command line). The DAQApplicationManager GUI is good for restarting assembler, DDS instances, etc.