Project

General

Profile


Quick Info

  • RUN COORDINATOR Minerba Betancourt (612-859-9078)
  • Google Hangouts ID: (password is the same as the sbn-doc-db)
  • Hangouts Group
  • VST shift: +1 331 442 4353 (Hangouts Voice Service)
  • ROC West x5413
  • MCenter Control Room (MC-CR) x3726 or x8871
  • Internationally: prefix +1 630 840 to any extensions (marked with x)
  • Experts list

Welcome to the SBND VST Wiki

This is the wiki for the SBND's Vertical Slice Test (VST). This is an integration of the cold and warm electronics with DAQ and online monitoring using LArIAT's cryostat and TPC.

HUGE DISCLAIMER: IT IS TOO SOON TO GET COMFORTABLE
THIS PAGE IS BEING DEVELOPED RIGHT NOW! CHECK IT EVERY TIME SO YOU ARE UPDATED TOO!

Run Plan for August 20 to September 05

  • At the start of your shift, please end the previous run by stopping the SBND DAQ and then the LArIAT DAQ, and start them both up again (first LArIAT, followed by SBND). Be sure to document this in the elog.
  • When the new run starts, the Online monitor show reliable information for the noise correlation matrix and the wires rms. Save a snapshot of

1) the Correlation Matrix
2) the Collection Plane RMS
3) the Collection Plane Pulse Height

and add them to a New Run e-log entry (see entry 297 as an example).

There are now 4 types of reports:

Start of Shift Checklist
End of Shift Checklist

New Run
End of Run

The shifter must fill in all of them. Instructions are in the form themselves.

At the end of the shift, shifters need to post an event display image of the data collected during the shift.

Taking VST Shifts

In summary, a VST shift is just like a LArIAT shift, with a few extras. If you didn't do this yet, make sure you are in the SBND.Shift group at Google Hangouts.

Prerequisites

Gaining Access

  • First, set up your Fermilab computing account and Kerberos. This can be done by following instructions here. Check that you can issue a Kerberos ticket with
 kinit <yourKerberosPrincipal>@FNAL.GOV

and confirm that you have gotten a kerberos ticket by checking the output of the command klist.

  • If you are at Fermilab:
    • Check that you are connected via ethernet, or to the Fermilab fgz Wi-Fi network and not the guest Wi-Fi network. This is required to access the lariat shift machines. The latter may lead to "Connection reset by peer" errors, refusing you access. Register your device to the fgz network here.
    • If you find you cannot SSH into lariat-cr-02.fnal.gov, make sure your SSH config file allows X11 forwarding and all GSSAPI options. Further details can be found here.
  • If you are outside of the lab:
    • The Java dependent Cisco AnyConnect client can no longer run from a browser due to the lack of support for NPAPI. This means that the old VPN method (https://vpn.fnal.gov) only works in old browsers. Use this as an alternative method, which is browser-independent:

Uninstall Cisco AnyConnect (skip this if you are sure you don't have it):

sudo /opt/cisco/vpn/bin/vpn_uninstall.sh

Install openconnect (it is an open source vpn client)

if you are using ubuntu...

sudo apt-get install openconnect 

if not, use the appropriate method for your system. You can verify the installation by asking the version:

openconnect --version

Install a vnc client of your choice. TigerVNC or Remmina work perfectly.

When you are about to start

Contact the current shifter in the hangouts group to tell him/her that you are getting ready. This will give the current shifter time to fill in his End Of Run report. Ask about the current plan and status (the shifter might ask you to read some e-log entries).

Don't log into the hangouts yourself just yet. Wait until the current shifter tells you that he/she is ready to give you the controls.

Setup your shift station

Either from a remote connection or from ROC-West, follow these instructions.

Log onto the shift computer

For a shift at Fermilab:

  • From a ROC-West machine, such as lariat-cr-02:
    • Log in with username = lariat.
  • From an MCenter machine, such as ftbflx18 or ftbflx15:
    • Log in with username = ftbf_user.
  • Password is YOUR kerberos password, then it will ask for YOUR kerberos principal. (Reverse from usual order)

From a remote shift:

  • VPN - To connect to the Fermilab VPN, open a terminal and execute:
sudo openconnect https://vpn.fnal.gov

and it is going to ask for your Username and Password (your SERVICES password).

If it works, you will be greeted with this message:

| NOTICE TO USERS
| This  is a Federal system (and/or it is directly connected to a Fermilab local network) ...

That means you are in. Do not close this terminal.

  • SSH Tunnel - In another terminal, get a Kerberos ticket
kinit <username>@FNAL.GOV

with your Kerberos ID. Now, open the tunnel with:

ssh -C -L 5900:localhost:5900 lariat@lariat-cr-02.fnal.gov

and you know it worked if you get another greeting:

Welcome to the LArIAT ROCwest Control Room

Again, leave this terminal open. If this fails, ask one of the following collaborators ( or ) to add you to the ~lariat/.k5login list, which will grant you SSH access.

  • VNC - Finally opening the VNC. You may use a client of you choice. When it asks for a server, use localhost:5900 which is our tunnel. The password is the same as the lariat-doc-db password (NOT SBND's).
  • Please do not interfere with the shifter by VNC! Contact the shifter on Google Hangouts to confirm a convenient time before you test your VNC.
  • Google Hangouts from Chrome browser (phone function does not work in Firefox)
    • Try to make a free phone call to a US number using the hangouts dialer (may require installing Google Talk plugin first time)
    • This is how the Main Control Room will phone you (it will ring when they dial +1 331-442-4353) and it is how you should phone them.

Windows needed on the shifter computer (or by VNC)

See these snapshots of the ROC-West desktop should give you a good idea about what to look for.


Verify that Synoptic and Acnet are running

You should get familiar with them. Always keep them open and in clear view. If connecting remotely, you should see them in the VNC shared desktop.

https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring

https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Using_ACNET_for_monitoring

Connect as the SBND shifter in the hangouts

When the previous shifter had time to call the MCR and declare the end of his/her shift, you may log into the shifter's Hangouts account with:

using the current SBND DocDB password. The SBND Shift chat is used by the collaborators to communicate with the shifters and between themselves about the VST operations.

Log into the SBND Electronic Logbook

http://dbweb0.fnal.gov/ECL/sbnd

with your Services user. Make an VST Start of Shift Cheklist entry and follow the checklist, which includes calling the Main Control Room (see next step).

Calling the Main Control Room (MCR)

Remember to call MCR (x3721) at the beginning and ending of each shift. If you are remote, you can use the hangouts to do so, but you may have to install a plugin first.

Main Control Room (internationally) +1.630.840.3721

Tell the MCR that you are the next LARIAT SHIFTER and if you are local or remote. If you are remote, inform the MCR of your contact number, which is the hangouts voice service:

+1 331 442 4354

You have to call the MCR at the begging and ending of each shift!

During your shift

See the RUN PLAN at the top of the page.

Either in your machine or in the one you are remotely operating via VNC, you have to keep 3 (three) independent terminals running: LArIAT DAQ, SBND DAQ and SBND DQM.

If for any reason you have to access those yourself (sometimes it happens when the previous Kerberos ticket expires), here is how:

  • With the tunnel open, log in to the lariat-gateway with the sbnd user:
kinit <youruser>@FNAL.GOV
ssh -X -Y sbnd@lariat-gateway.fnal.gov

once you are in the gateway, access lariat-daq00 if you want to launch the LArIAT DAQ, or lariat-daq04 for the SBND DAQ (you want both, so repeat this procedure twice).

kinit <youruser>@FNAL.GOV
ssh -X -Y sbnd@lariat-daq04.fnal.gov

again, substitute daq04 with daq00 if you want to launch the LArIAT DAQ.

In the DAQ00 window, execute the LArIAT script:

go

Do not specify any arguments.

In the daq04 machine, excecute the SBND script:

startDAQInterface

You also need the SBND DQM website (which can only see this via Fermilab VPN or locally through the fgz network): https://sbn-online.fnal.gov/cgi-bin/minargon/minargon.wsgi
If you are in a remote shift, you can open another terminal and make use of the tunnel. Launch your favorite browser from the terminal, for instance

google-chrome

and it will be able to access the DQM site.

Keep checking the Synoptics alarms and the DQM website. The DQM has several plots that should be updating every few minutes. If you see any of the time stamps getting too old, you may need to restart it (instructions above). Instructions on how the DQM works are in its first page.

Archiver checks
Once an hour, check the status of the archiver by following the instructions on the Slow Controls Archiving page.

Some Troubleshooting

If the SBND DAQ won't start, there are two steps to follow, in this order:

Step 1. Try again: sometimes it just need some persistence.

Step 2. Other times, it might need some clean up. Try running this script:

kill_ipcs

and go back to Step 1.

Another problem is data corruption. You will see the RMS of the Wires going through the roof. If that happens, try this one first without turning off the daq.

source start_bnl_ld.sh
python WIB_nevis_sync.py

After about 3 minutes, you should see the rms going back to normal. If not, then follow this recipe:

stop the daq (ctrl+C only once)

Reconfigure the cold electronics
Note: this is the standard CE configuration procedure, so can be used for any time configuration is needed. If starting from state where electronics are off, you can skip the FEMBoff step.

ssh lariat-gateway02
source start_bnl_ld.sh 
source Local_diagnositc_RUN_FEMBoff.sh 
source Local_diagnositc_RUN_FEMBon.sh 
source Local_diagnositc_RUN_ext.sh 

start the daq on lariat-daq04

startDAQInterface

Report any oddities in the e-log with a default form, under the Vertical Slice Test category.

Event Display

There are two ways to the run the event display now. If you want to look at the most up to date data. Data which might not have gone through SAM yet you can log into the daq04 machine and use the event display there. Otherwise you can you the event display on the sbnd gpvms.

daq04 Event Display.

To run the event display. From the control room PC log into daq04:

ssh -X lariat-gateway -l sbnddqm -t ssh -X lariat-daq04

Then run the command replacing the run number you wish to look at and the sub run number:

 
./home/nfs/sbnd/SBND_DAQ2/bin/evd_vst  run_number sub_run_number

This will bring up the standard event display which you can flick through the events.

SBND GPVM Event Display.

log into the sbndgpv01. Remember to kinit (kinit ).

ssh username@sbndgpvm01.fnal.gov. 

Then got the VST directory in app just running the following command:

cd /sbnd/app/users/VST
bash evd_vst.sh

This will cycle through the events and automatically pick up and read the latest file. You can pause the cycle by typing PAUSE (upper or lower case) into the command line and pressing enter (you will not see that you are writing pause). You can then save the event display. Type PLAY (upper or lower case) to continue the loop. NOTE: If you press previous on the GUI then when you press play it will loop backwards. So press next once to sort the loop out. Note that I've only just made it so it might be a little buggy. If you see an error about trying to write to a read-only database then someone is already running the event display in there. Contact Dom Barker for issues.




Advanced Stuff

How to change the DAQ configuration to calibration mode

At lariat-daq04.fnal.gov, open the file /home/nfs/sbnd/SBND_DAQ2/bin/start_daq_interface.sh and comment the line

CONFIG=NevisTPCXMIT30 # External triggers

and uncomment the line

#CONFIG=NevisTPCCalib # Internal calibration triggers

Then you can start a new run following the usual procedure. This will run the DAQ using internal triggers and a calibration pulse will be sent to the Magic Blue Box for each trigger.

When you are done, remember to undo the changes to the /home/nfs/sbnd/SBND_DAQ2/bin/start_daq_interface.sh file.

How to run the DAQ for the BNL-Nevis integration test

(Note: this DAQ routine has been superseded by How to change the DAQ configuration to calibration mode . If you want to run the DAQ in calibration mode, follow the instructions there)

1. Make sure the Nevis crate and NIM bin are powered. Experts: (on-site) Davio Cianci, (off-site) José I. Crespo (Skype: josei.ca)
2. ssh into the DAQ machine
Log into the SBND gateway (or you can create an FNAL VPN connection and skip sbnd-daq00, not needed on FNAL site)

ssh -Y sbnd-daq00.fnal.gov

Log into the LArIAT gateway

ssh -Y lariat-gateway01.fnal.gov

Log into the DAQ server using the common sbnd account (reserved for DAQ)

ssh -Y sbnd@lariat-daq04

3. Go to the jcrespo/ directory

cd jcrespo

Run the DAQ-decoder-plotter script

./run_and_plot_sbnddaq.sh

and follow the instructions printed on screen. This script runs the DAQ in calibration mode (CALIB pulses are sent to the Magic Blue Box every 1 s) -- once you have collected the desired number of events, it requires you to stop the DAQ using CTRL + C twice --, then runs the binary decoder, and finally runs a ROOT analysis to plot the waveforms.

The data is saved in the directory /home/nfs/sbnd/jcrespo/runs
The script makes a new directory for each run named RunYearMonthDayHourMinutesSecond (e.g. Run20180620113511).
The raw binary data is the file named output_NevisTPCCALIB_generator.dat. For a hexdump, you can use

hexdump -v -e '8/2 "%04X ""\n"' output_NevisTPCCALIB_generator.dat | less

(it will display the data in 8 columns with 2 bytes each). The ADC data is encoded in the lower 12 bits (3 last hex characters) of the words beginning with "0".

How to configure LArIAT Trigger

Follow the link LArIAT Trigger

How to Access Files

All files created for the VST test are initially created on lariat daq01 and daq04. They are despoiled in /daqdata and are quickly moved to /daqdat/dropbox where they last for ~2 days until they are moved to SAM. Files beginning with sbnd are the raw data non-decoded files. Files beginning with digits_sbnd are larsoft friendly files. These have been decoded and raw:digits objects have been created. Files beginning with lariat for the corresponding runs contain the beam-line detector information.

Raw Data files starting with sbnd from runs prior to Thursday 13th July are now temporarily being stored in /pnfs/sbnd/persistent/users/VST/sbnd_files. You can access these from the SBND GPVM. Gray is putting the digits on pnfs as well. Soon there will be the corresponding digits file as well.

All Other files can be found on SAM. All the files come under the lariat experiment name so make sure there is a "-e lariat" in the commands you write. To Quickly access the files here is a set of instructions:

Setup the required stuff.

source /grid/fermiapp/products/setups.sh; 
setup ifdhc; 
setup sam_web_client; 
setup fife_utils; 

To Look at the files available you can use some of the SAM commands. Here is a few examples:

To see a list of files from after a date:

samweb -e lariat list-files "start_time > 02-Jul-2018" 

To see all the files which have sbnd:

samweb -e lariat list-files "file_name like sbnd_r%.root" 

To use metadata.

samweb list-files -e lariat "MetaDataName = value"

Use And to do two or more filters :

samweb -e lariat list-files "start_time > 02-Jul-2018" AND file_name like sbnd_r%.root 

You can look at the MetaData in the files by running:

samweb -e lariat get-metadata file.root

You can locate the file on enstore:

samweb -e lariat locate-file file.root 

To use the files you have to stage them and copy them into your directory. This can be done using project.py as a wrapper once definitions of the data have been created.
For now you can use ifdh_fetch to fetch one or more files. It works in the same way as SAM and here is a couple examples on using it:

Know the file name:

ifdh_fetch -e lariat filename.root

Copy every file with a name that begins with sbnd_r. (NOTE: We have the downloaded so don't do it again. Just run on them. If they have been deleted because it has been a long time and we need the space for MC productions then you can download them but make sure we have the space!):

ifdh_fetch -e lariat --dims "file_name like sbnd_r%.root"

For more sbnd specific information you can find another wiki here: https://cdcvs.fnal.gov/redmine/projects/sbndcode/wiki/_How_to_create_and_use_a_SAM_definition.