THIS PAGE HAS MOVED TO the DUNE wiki
Request access to the wiki by sending an email to dune-communication@fnal.gov.
- Table of contents
- Getting Started with DUNE Computing
- Getting Accounts and Logging In at Fermilab
- Kerberos Tips and Info
- UPS Tips and Info
- General Overviews
- Some Exercises to Help You Get Started
- Log in to a DUNE interactive node
- Some Useful Commands to Work Through To Setup Your Environment
- Run Simulation and Reconstruction
- Copy a File to dCache Mass Storage
- Execute SAMweb Commands
- Run an Event Display
- Look at the FIFE Monitoring
- Extra Credit: Looking Around
- Example: Building a New Analysis Module Based on larexamples
Getting Started with DUNE Computing¶
THIS PAGE HAS MOVED TO the DUNE wiki
Getting Accounts and Logging In at Fermilab¶
THIS PAGE HAS MOVED TO the DUNE wiki
Request access to the wiki by sending an email to dune-communication@fnal.gov.
You must be on the DUNE Collaboration member list in order to get an account (check if you are on it at-): https://dune.bnl.gov/people/
Talk to your Institutional Board representative to get on it. A list of representatives is at https://dune.bnl.gov/people/members/export/?filename=iblist.txt&date=now
Instructions for getting accounts: https://web.fnal.gov/collaboration/DUNE/SitePages/Getting%20Computer%20Accounts%20at%20Fermilab.aspx
Fermilab uses Kerberos to implement strong authentication (no passwords on the internet) when logging in to Fermilab machines.
Make sure the Fermilab KDC's are in your /etc/krb5.conf file.
Make sure your ~/.ssh/config file has default login options like the following -
ForwardX11 yes ForwardX11Trusted yes GSSAPIAuthentication yes GSSAPIDelegateCredentials yes
The following example shows how to log in from a Mac with kerberos and ssh installed -
my-mac$ type kinit kinit is /usr/bin/kinit # case matters for FNAL.GOV, login using your Kerberos password kinit -a -r7d –f your-kerberos-principal@FNAL.GOV # this is the OpenSSH client my-mac$ type ssh ssh is hashed (/usr/bin/ssh) ssh -K your-username@dunegpvm01.fnal.gov
Kerberos Tips and Info¶
THIS PAGE HAS MOVED TO the DUNE wiki
Request access to the wiki by sending an email to dune-communication@fnal.gov.
Kerberos tickets (what you get with the kinit command) have a default lifetime of 26 hours after which they expire. If you use the -r option on the kinit line, then your ticket can be renewed instead of having to get a new one.
Users must have a valid kerberos ticket to access Fermilab computing at the time an attempt to log into a Fermilab machine. The ticket is obtained by executing the following command at a terminal prompt:
$ kinit <your_Kerberos_principal>@FNAL.GOV
where <your_Kerberos_principal> is the user's kerberos principal (i.e., username or uid). If a user is attempting to access the repository from a non-Fermilab machine, the following lines must be in the user's .ssh/config:
Host *.fnal.gov
ForwardAgent yes
ForwardX11 yes
ForwardX11Trusted yes
GSSAPIAuthentication yes
GSSAPIDelegateCredentials yes
It is possible to allow other users (or yourself just on another machine or with another Kerberos identity) to access your account via a .k5login file in your $HOME directory. A warning however: If you create a .k5login file, make sure you put your own username in it or you can be locked out of your own account. It should have the line
<your_Kerberos_principal>@FNAL.GOV
in it.
Users logging in from outside the Fermilab network may be behind Network Address Translation (NAT) routers. If so, you may need an "addressless" ticket. Experiment with -a and -A options to kinit. Some users have reported needing to specify the -K option to ssh in order to get the GSSAPI Authentication to work.
Some users have reported problems with the Kerberos utilities provided by Macports. Macintosh users should use the system-provided Kerberos utilities -- like /usr/bin/kinit.
Additional help (if you want to know more or need to troubleshoot) -- useful tips on logging in with Kerberos:
https://fermi.service-now.com/kb_view_customer.do?sysparm_article=KB0011308
https://fermi.service-now.com/kb_view_customer.do?sysparm_article=KB0011313
and an introductory explanation of tickets, certificates, and proxies is available at:
http://cd-docdb.fnal.gov/cgi-bin/ShowDocument?docid=5892
Some links which might be helpful for using non-Fermilab-managed Windows systems. These instructions have
not been tried by the authors of this wiki.
https://fermi.service-now.com/kb_view_customer.do?sysparm_article=KB0011314
and some help with Redmine for Windows users: https://cdcvs.fnal.gov/redmine/projects/fermi-redmine/wiki/Windows
UPS Tips and Info¶
THIS PAGE HAS MOVED TO the DUNE wiki
Request access to the wiki by sending an email to dune-communication@fnal.gov.
UPS is environment management software for handling software products with many versions and different 'flavors' of components. You use it to make sure you are using the correct version of the product you need and any dependent products that one may rely on.
In the table below the following is true -- dunetpc - the product being setup
- v_06_34_00 - the version of the dunetpc product for this example. This may not be the latest version.
- e14:prof - the qualifiers for the version of dunetpc. Qualifiers further define something about the version of dunetpc, like how it was built (with profiling or prof) or the version of the gcc compiler used (e14) in this example. You an find out what the qualifier e14 and similar ones mean here .
Basic Command Information¶
These commands are done after logging in to the DUNE interactive nodes.
Command | Description | Use |
---|---|---|
source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh | Configure your environment, get access to software versions | Use once at login |
setup -B dunetpc v06_34_00 -q +e14:+prof | Set up a particular version of dunetpc and all dependencies | Use once after login and after running setup_dune.sh |
ups list -aK+ dunetpc | Find out which versions and flavors of dunetpc exist on this node | Use whenever you need to find out what's available |
ups active | Find out what has been setup | Use when you want to |
ups depend dunetpc v06_34_00 -q +e14:+prof | Find out what depends on what for this version of dunetpc | Use when you want to |
After doing the setup of dunetpc you can see where the software is by looking at the DUNETPC_DIR (<PRODUCT>_DIR>) variable.
$ echo $DUNETPC_DIR /cvmfs/dune.opensciencegrid.org/products/dune/dunetpc/v06_34_00
Further Documentation¶
When you need to learn more about UPS, visit the following links.
Info on Qualifiers: https://cdcvs.fnal.gov/redmine/projects/cet-is-public/wiki/AboutQualifiers
UPS Full Documentation: http://www.fnal.gov/docs/products/ups/ReferenceManual/
General Overviews¶
Info on many of the tools you will use - Getting Started With FIFE Tools¶
Read the landing page to learn a little about art¶
Read an intro about gallery¶
Some Exercises to Help You Get Started¶
Log in to a DUNE interactive node¶
Instructions to log in to dunegpvm01 using ssh are in the above section on Getting Accounts and Logging In at Fermilab.
Some Useful Commands to Work Through To Setup Your Environment¶
Run these commands on the DUNE interactive node after you have logged in. They will show you some information and get the environment setup for the rest of the exercises. You will only have to do these once.
Command | Description |
---|---|
df -h | find out what disks are connected to the node and their capabilities |
pwd | what directory are you in |
cd | change directories to your home area |
ls | look at the files in your directory |
echo $USER | make sure the USER variable is set |
export USER=`whoami` | set the USER variable if not already set |
ls /dune/app/users/$USER | Check to see if you have directories in the user area |
mkdir /dune/app/users/$USER | If it does not exist, make your directory on the app disk |
mkdir /dune/data/users/$USER | If it does not exist, make your directory on the BlueArc data disk (mounted no execute) |
mkdir /dune/data2/users/$USER | If it does not exist, make your directory on the 2nd BlueArc data disk (mounted no execute) |
mkdir /pnfs/dune/scratch/users/$USER | If it does not exist, make your directory on the scratch dCache disk |
mkdir /pnfs/dune/persistent/users/$USER | If it does not exist, make your directory on the persistent dCache disk |
Run Simulation and Reconstruction¶
Execute the following commands after logging in to the DUNE interactive node and creating the directories above -
cd /dune/data/users/$USER mkdir example_11_2017 cd example_11_2017 # setup the dunetpc environment source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh # see what versions of dunetpc there are ups list -aK+ dunetpc # setup a recent one on the list: setup -B dunetpc v06_55_01 -q e14:prof # ups automatically sets up all the packages dunetpc depends on. Show a list a list of active packages: ups active # the main executable program is called "lar". Find out where it is: which lar # root is also set up which root # run the generator for a single muon in the 1x2x6 far detector workspace geometry # all lar commands, is successful, will end with a line - Art has completed and will exit with status 0. lar -n 1 -c prod_muminus_0.1-5.0GeV_isotropic_dune10kt_1x2x6.fcl # see what you got. It's just the generator, which makes mctruth and mcparticle data ls -lrt # run GEANT4 on the output of the generator made above. lar -c standard_g4_dune10kt_1x2x6.fcl prod_muminus_0.1-5.0GeV_isotropic_dune10kt_1x2x6_gen.root # see what you got ls -lrt # run the detector simulation on the G4 output. # This generates waveforms on the wires, adds noise, digitizes and zero-suppresses the output lar -c standard_detsim_dune10kt_1x2x6.fcl prod_muminus_0.1-5.0GeV_isotropic_dune10kt_1x2x6_gen_g4.root # see what you got. Newest file is on the bottom ls -lrt # run the standard reco on the output of the detsim stage lar -c standard_reco_dune10kt_1x2x6.fcl prod_muminus_0.1-5.0GeV_isotropic_dune10kt_1x2x6_gen_g4_detsim.root
Copy a File to dCache Mass Storage¶
Execute the following commands after logging in to the DUNE interactive node, creating the directories above and executing the Simulation and Reconstruction commands -
# copy the output file to dCache scratch space ifdh cp -D prod_muminus_0.1-5.0GeV_isotropic_dune10kt_1x2x6_gen_g4_detsim_reco.root /pnfs/dune/scratch/users/$USER/
Execute SAMweb Commands¶
This exercise will start you accessing data files that have been defined to the DUNE Data Catalog.
Execute the following commands after logging in to the DUNE interactive node, creating the directories above -
mkdir samweb cd samweb samweb locate-file lbne_r004024_sr01_20151027T154934.root
this will give you output that looks like
enstore:/pnfs/lbne/test-data/lbne/raw/00/04/41/51(3029@vpl189)
which is the location of the file on tape. We can use this to copy the file from tape to our local disk.
NOTE: we are on the data disk (/dune/data/users/...), you should never store data files on the app disk (/dune/app/users/...).
ifdh cp -D /pnfs/lbne/test-data/lbne/raw/00/04/41/51/lbne_r004024_sr01_20151027T154934.root .
To get SAM metadata for a file for which you know the full name:
sam_metadata_dumper /pnfs/lbne/test-data/lbne/raw/00/04/41/51/lbne_r004024_sr01_20151027T154934.root
A more complete metadata printout can be had with this command:
samweb -e dune get-metadata lbne_r004024_sr01_20151027T154934.root
To list raw data files for a given run:
samweb list-files "run_number=4024 and data_tier=raw"
Run an Event Display¶
IMPORTANT FIRST STEP : Follow the instructions for getting VNC running - https://cdcvs.fnal.gov/redmine/projects/dune/wiki/Using_VNC_Connections_on_the_dunegpvms
Execute the following commands after logging in to the DUNE interactive node, creating the directories above and executing the Simulation and Reconstruction commands -
# run the following command to test your connection. If nothing appears on your vnc display, the setup is not correct xlogo # run the event display. You will have to click through the TPC selector button to find your particle: lar -c evd_dune10kt.fcl prod_muminus_0.1-5.0GeV_isotropic_dune10kt_1x2x6_gen_g4_detsim_reco.root
Look at the FIFE Monitoring¶
The FIFE Monitoring displays statistics on jobs, storage uses etc. This can be a useful place to look if things are not happening as you expect -
https://fifemon.fnal.gov/monitor/dashboard/db/experiment-overview?var-experiment=dune
You will be prompted to log in with your Fermilab services username and password.
Click on the graphs to see what you can do.
Extra Credit: Looking Around¶
Execute the following commands after logging in to the DUNE interactive node and running the setup commands above -
# There are four colon-separated search paths for finding stuff. They are built with the setup command above # For commands: echo $PATH # for shared libraries: echo $LD_LIBRARY_PATH # for fcl files (which steer a job) echo $FHICL_FILE_PATH # for other files that the framework needs (like GDML files for geometry, or photon libraries): echo $FW_SEARCH_PATH # ups sets up environment variables per package that point to the locations of the installed products echo $DUNETPC_DIR # subdirectories in there include source code, headers, pre-built and installed libraries, and fcl files # look at them all! env | grep DIR
Example: Building a New Analysis Module Based on larexamples¶
We will be coaching users through this during the afternoon session at the tutorial. Follow the link below for instructions.