Project

General

Profile

RC - NewGuide

This page contains basic instructions on what to do for normal DAQ operations. If you are having a problem, you can look here to see what you should do, but you should also consider looking at the RC - Troubleshooting page, which contains more details of common problems.

Your job, as the shifter

First, THANK YOU! In taking your shift on MicroBooNE, you are the person/people that is taking the data from the detector (and doing a lot more!). That makes you part of the DAQ team---in fact, a very crucial part---so, welcome, and thank you for agreeing to help us.

The parts below explain how to do the basic DAQ tasks in more detail, but at a high-level view, your basic responsibilities as a member of the DAQ team are:
  • Start and stop the data acquisition processes, and make sure they stay running throughout your shift, according to the Run Plan.
  • Watch the DAQ processes to make sure we are taking data smoothly (i.e. that we are taking data, and not automatically starting new runs all the time).
  • Note in the e-log the state of the DAQ, any problems/strange things that come up, or any other detector conditions that may (or may not) affect data-taking (e.g. accesses to the LArTF platform!)
  • Promptly notify more-experienced members of our DAQ team when you notice a problem.
  • Help more-experienced members of the DAQ team solve problems by describing and documenting problems you encounter in the e-log, and maintaining awareness throughout the data-taking so that you can help answer questions those more-experienced members of the DAQ team may have.
  • Provide feedback on how we can make the DAQ experience better for future shifters like you.

It's not part of your job description to try to fix problems yourself. We will talk about basic trouble-shooting and recovery techniques you should do when we encounter problems, but beyond those things, it's not your job to try to fix something, and you are not responsible for any downtime. We need you to vigilantly watch, report, and call for help!

Getting help

There are a number of ways that you can contact experts: the Slack chat, email, carrier pigeon, skype, phone, etc. Luckily/not luckily, the DAQ team is spread across a number of time zones, so it's likely that no matter the time of day, somebody might even be watching and ready to jump in to help. How you get help is not as important as making sure you do, and quickly, especially when we are getting beam data. To make sure you get help quickly though, here's what we recommend doing when you notice a problem:
  1. Put a quick message on the Slack chat like "Need DAQ expert!" or "I don't know if I'm having a problem, can someone help?"
  2. Don't wait long for a reply, especially if it's keeping us from taking beam data or you think it may be. Maybe wait just long enough for you to put a quick note into the e-log or take another glance at the trouble-shooting page, but probably no longer than 30 seconds to a minute. Seriously, don't wait before you ...
  3. Call the on-call DAQ expert. That person is on-call 24/7 and should be reachable at all times. Please CALL FROM THE SHIFT PHONE so experts can see the Control Room is calling. If you get voicemail, leave a short message saying you are having a problem (or may be having a problem), and note the time.
  4. If you didn't catch them on the phone, don't wait long for a reply. Like, 5-7 minutes max. Keep trying to restart runs, note in the elog the problem in more detail, send another quick message on the Slack chat, etc. Seriously, don't wait longer than a few minutes before you ...
  5. Call other experts. Maybe try the on-call expert again, but if no answer immediately go down the DAQ expert list and try to get someone on the phone! If no one is answering, and you're not getting any help anywhere, then call the Run Coordinator, who can advise on other experts to call or help get in contact with DAQ experts. You should also be putting passive-aggressive comments in the elog to publicly shame the DAQ experts.

Starting the DAQ VNC

If it is not open, connect to the vnc session for the DAQ by opening a terminal and doing:

ssh -L 5908:localhost:5905 uboonedaq@ubdaq-prod-ws01.fnal.gov

(you may need to refresh the kerberos before doing this)

Now, start the vnc session:

vncviewer localhost:5908

The password is the common DAQ password xxxxxxxxxxx (concatenate the MicroBooNE DocDB user name, the number two, and the first syllable in the word preceding "xxxxxxxxxxx" in this sentence). This should open you up to a desktop on ubdaq-prod-ws01 as uboonedaq

Starting a run

In the vnc session (see above item), open a terminal and log in to ubdaq-prod-evb.fnal.gov (still as uboonedaq):

ssh evb

You should see a bunch of setup things reported. You should see in blue text the DAQ version number. If a DAQ expert wants you to change versions, all you should need to do is log out of evb, and log back in to get the desired one.
Do "runConsoleDAQ.py" on terminal:
runConsoleDAQ.py

You will then be prompted to enter how long you would like this run to be. You cannot put a value in greater than 420 minutes. You can exit the console daq script here by typing 'exit'. Consult the white board, elog, run plan, or run coordinator on what should be entered here (but for normal data-taking, 420 minutes is good).

After that, you will be prompted to pick a configuration ID. Consult the white board, elog, run plan, or Run Coordinator for the config ID we want to run. There are semi-descriptive names that go along with the ID, but you will need to enter the ID number on the command line.

Starting an expert run

Note, if you are doing a special configuration (and someone should tell you when we are doing so), you will need to run this in "expert" mode. How? With the expert option of course!

runConsoleDAQ.py expert

And, if it's a pmt only run, it only make sense to do...
runConsoleDAQ.py expert pmtonly

"Expert" mode just allows additional configurations, but has no auto-restart, so you will need to be more vigilant. PMTONLY mode does not read out TPC data, so will only start one seb process (that on seb10).

Stopping a run...

...while taking data

As the runConsoleDAQ should tell you, to stop a run while we are taking data, just hit [ENTER] in the runConsoleDAQ window. You will be prompted for a reason (that will be put into the elog), and you will be brought back to the start to begin a new run. The next run will not start automatically, so you will need to input a time and configuration again.

...at any other time (like during configuration)

If you are not in the middle of a run, you can "CTRL-C" in the runConsoleDAQ window, and it should kill all the processes cleanly. You will likely be prompted for a reason for the stop/cancel of the run, and that will automatically be put in the elog. After this, runConsoleDAQ will likely be killed, and you will need to run it again.

Cleaning up the DAQ.

There may be occasions where we have tried to stop a run, but for some reasons all processes have not been cleanly killed. For instance: you have exited out of runConsoleDAQ, but for some reason there are some blue xterm windows (from the assembler or seb) still displaying. To clean up, do:

terminate-online-daq-prod

Viewing the ganglia monitoring page

On one of the control machines, type the following command:

ssh -D 8881 uboonedaq@ubdaq-prod-ws01.fnal.gov

(you may need to renew your kerberos ticket).

Then, on that same machine, you should be able to navigate to the ganglia web page: http://ubdaq-prod-evb.fnal.gov:8080/gweb/

The username is uboone, and the password is the common DAQ password xxxxxxxxxxx (concatenate the MicroBooNE DocDB user name, the number two, and the first syllable in the word preceding "xxxxxxxxxxx" in this sentence).