Project

General

Profile

History

Wiki » Computing How-To Documentation »

THIS PAGE HAS MOVED TO the DUNE wiki

Request access to the wiki by sending an email to .

Getting Started with DUNE Computing

THIS PAGE HAS MOVED TO the DUNE wiki

Getting Accounts and Logging In at Fermilab

THIS PAGE HAS MOVED TO the DUNE wiki

Request access to the wiki by sending an email to .

You must be on the DUNE Collaboration member list in order to get an account (check if you are on it at-): https://dune.bnl.gov/people/
Talk to your Institutional Board representative to get on it. A list of representatives is at https://dune.bnl.gov/people/members/export/?filename=iblist.txt&date=now

Instructions for getting accounts: https://web.fnal.gov/collaboration/DUNE/SitePages/Getting%20Computer%20Accounts%20at%20Fermilab.aspx

Fermilab uses Kerberos to implement strong authentication (no passwords on the internet) when logging in to Fermilab machines.
Make sure the Fermilab KDC's are in your /etc/krb5.conf file.

Make sure your ~/.ssh/config file has default login options like the following -

ForwardX11 yes
ForwardX11Trusted yes
GSSAPIAuthentication yes
GSSAPIDelegateCredentials yes

The following example shows how to log in from a Mac with kerberos and ssh installed -

my-mac$ type kinit
kinit is /usr/bin/kinit

# case matters for FNAL.GOV, login using your Kerberos password
kinit -a -r7d –f your-kerberos-principal@FNAL.GOV 

# this is the OpenSSH client
my-mac$ type ssh
ssh is hashed (/usr/bin/ssh) 

ssh -K your-username@dunegpvm01.fnal.gov

Kerberos Tips and Info

THIS PAGE HAS MOVED TO the DUNE wiki

Request access to the wiki by sending an email to .

Kerberos tickets (what you get with the kinit command) have a default lifetime of 26 hours after which they expire. If you use the -r option on the kinit line, then your ticket can be renewed instead of having to get a new one.

Users must have a valid kerberos ticket to access Fermilab computing at the time an attempt to log into a Fermilab machine. The ticket is obtained by executing the following command at a terminal prompt:

$ kinit <your_Kerberos_principal>@FNAL.GOV

where <your_Kerberos_principal> is the user's kerberos principal (i.e., username or uid). If a user is attempting to access the repository from a non-Fermilab machine, the following lines must be in the user's .ssh/config:

Host *.fnal.gov
ForwardAgent yes
ForwardX11 yes
ForwardX11Trusted yes
GSSAPIAuthentication yes
GSSAPIDelegateCredentials yes

It is possible to allow other users (or yourself just on another machine or with another Kerberos identity) to access your account via a .k5login file in your $HOME directory. A warning however: If you create a .k5login file, make sure you put your own username in it or you can be locked out of your own account. It should have the line

<your_Kerberos_principal>@FNAL.GOV

in it.

Users logging in from outside the Fermilab network may be behind Network Address Translation (NAT) routers. If so, you may need an "addressless" ticket. Experiment with -a and -A options to kinit. Some users have reported needing to specify the -K option to ssh in order to get the GSSAPI Authentication to work.

Some users have reported problems with the Kerberos utilities provided by Macports. Macintosh users should use the system-provided Kerberos utilities -- like /usr/bin/kinit.

Additional help (if you want to know more or need to troubleshoot) -- useful tips on logging in with Kerberos:

https://fermi.service-now.com/kb_view_customer.do?sysparm_article=KB0011308

https://fermi.service-now.com/kb_view_customer.do?sysparm_article=KB0011313

and an introductory explanation of tickets, certificates, and proxies is available at:

http://cd-docdb.fnal.gov/cgi-bin/ShowDocument?docid=5892

Some links which might be helpful for using non-Fermilab-managed Windows systems. These instructions have
not been tried by the authors of this wiki.

https://fermi.service-now.com/kb_view_customer.do?sysparm_article=KB0011314

and some help with Redmine for Windows users: https://cdcvs.fnal.gov/redmine/projects/fermi-redmine/wiki/Windows

UPS Tips and Info

THIS PAGE HAS MOVED TO the DUNE wiki

Request access to the wiki by sending an email to .

UPS is environment management software for handling software products with many versions and different 'flavors' of components. You use it to make sure you are using the correct version of the product you need and any dependent products that one may rely on.

In the table below the following is true -
  • dunetpc - the product being setup
  • v_06_34_00 - the version of the dunetpc product for this example. This may not be the latest version.
  • e14:prof - the qualifiers for the version of dunetpc. Qualifiers further define something about the version of dunetpc, like how it was built (with profiling or prof) or the version of the gcc compiler used (e14) in this example. You an find out what the qualifier e14 and similar ones mean here .

Basic Command Information

These commands are done after logging in to the DUNE interactive nodes.

Command Description Use
source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh Configure your environment, get access to software versions Use once at login
setup -B dunetpc v06_34_00 -q +e14:+prof Set up a particular version of dunetpc and all dependencies Use once after login and after running setup_dune.sh
ups list -aK+ dunetpc Find out which versions and flavors of dunetpc exist on this node Use whenever you need to find out what's available
ups active Find out what has been setup Use when you want to
ups depend dunetpc v06_34_00 -q +e14:+prof Find out what depends on what for this version of dunetpc Use when you want to

After doing the setup of dunetpc you can see where the software is by looking at the DUNETPC_DIR (<PRODUCT>_DIR>) variable.

$ echo $DUNETPC_DIR
/cvmfs/dune.opensciencegrid.org/products/dune/dunetpc/v06_34_00

Further Documentation

When you need to learn more about UPS, visit the following links.

Info on Qualifiers: https://cdcvs.fnal.gov/redmine/projects/cet-is-public/wiki/AboutQualifiers
UPS Full Documentation: http://www.fnal.gov/docs/products/ups/ReferenceManual/

General Overviews

Info on many of the tools you will use - Getting Started With FIFE Tools

Read the landing page to learn a little about art

Read an intro about gallery

Some Exercises to Help You Get Started

Log in to a DUNE interactive node

Instructions to log in to dunegpvm01 using ssh are in the above section on Getting Accounts and Logging In at Fermilab.

Some Useful Commands to Work Through To Setup Your Environment

Run these commands on the DUNE interactive node after you have logged in. They will show you some information and get the environment setup for the rest of the exercises. You will only have to do these once.

Command Description
df -h find out what disks are connected to the node and their capabilities
pwd what directory are you in
cd change directories to your home area
ls look at the files in your directory
echo $USER make sure the USER variable is set
export USER=`whoami` set the USER variable if not already set
ls /dune/app/users/$USER Check to see if you have directories in the user area
mkdir /dune/app/users/$USER If it does not exist, make your directory on the app disk
mkdir /dune/data/users/$USER If it does not exist, make your directory on the BlueArc data disk (mounted no execute)
mkdir /dune/data2/users/$USER If it does not exist, make your directory on the 2nd BlueArc data disk (mounted no execute)
mkdir /pnfs/dune/scratch/users/$USER If it does not exist, make your directory on the scratch dCache disk
mkdir /pnfs/dune/persistent/users/$USER If it does not exist, make your directory on the persistent dCache disk

Run Simulation and Reconstruction

Execute the following commands after logging in to the DUNE interactive node and creating the directories above - 

cd /dune/data/users/$USER

mkdir example_11_2017
cd example_11_2017

# setup the dunetpc environment
source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh

# see what versions of dunetpc there are
ups list -aK+ dunetpc

# setup a recent one on the list:

setup -B dunetpc v06_55_01 -q e14:prof

# ups automatically sets up all the packages dunetpc depends on. Show a list a list of active packages:
ups active

# the main executable program is called "lar".  Find out where it is:
which lar

# root is also set up
which root

# run the generator for a single muon in the 1x2x6 far detector workspace geometry
# all lar commands, is successful, will end with a line - Art has completed and will exit with status 0.
lar -n 1 -c prod_muminus_0.1-5.0GeV_isotropic_dune10kt_1x2x6.fcl

# see what you got.  It's just the generator, which makes mctruth and mcparticle data
ls -lrt

# run GEANT4 on the output of the generator made above.
lar -c standard_g4_dune10kt_1x2x6.fcl prod_muminus_0.1-5.0GeV_isotropic_dune10kt_1x2x6_gen.root

# see what you got
ls -lrt

# run the detector simulation on the G4 output.
# This generates waveforms on the wires, adds noise, digitizes and zero-suppresses the output
lar -c standard_detsim_dune10kt_1x2x6.fcl prod_muminus_0.1-5.0GeV_isotropic_dune10kt_1x2x6_gen_g4.root

# see what you got.  Newest file is on the bottom
ls -lrt

# run the standard reco on the output of the detsim stage
lar -c standard_reco_dune10kt_1x2x6.fcl prod_muminus_0.1-5.0GeV_isotropic_dune10kt_1x2x6_gen_g4_detsim.root

Copy a File to dCache Mass Storage

Execute the following commands after logging in to the DUNE interactive node, creating the directories above and executing the Simulation and Reconstruction commands -

# copy the output file to dCache scratch space
ifdh cp -D prod_muminus_0.1-5.0GeV_isotropic_dune10kt_1x2x6_gen_g4_detsim_reco.root /pnfs/dune/scratch/users/$USER/

Execute SAMweb Commands

This exercise will start you accessing data files that have been defined to the DUNE Data Catalog.
Execute the following commands after logging in to the DUNE interactive node, creating the directories above -

mkdir samweb
cd samweb
samweb locate-file lbne_r004024_sr01_20151027T154934.root

this will give you output that looks like
enstore:/pnfs/lbne/test-data/lbne/raw/00/04/41/51(3029@vpl189)
which is the location of the file on tape. We can use this to copy the file from tape to our local disk.
NOTE: we are on the data disk (/dune/data/users/...), you should never store data files on the app disk (/dune/app/users/...).
ifdh cp -D /pnfs/lbne/test-data/lbne/raw/00/04/41/51/lbne_r004024_sr01_20151027T154934.root .

To get SAM metadata for a file for which you know the full name:
sam_metadata_dumper /pnfs/lbne/test-data/lbne/raw/00/04/41/51/lbne_r004024_sr01_20151027T154934.root

A more complete metadata printout can be had with this command:
samweb -e dune get-metadata lbne_r004024_sr01_20151027T154934.root

To list raw data files for a given run:
samweb list-files "run_number=4024 and data_tier=raw"

Run an Event Display

IMPORTANT FIRST STEP : Follow the instructions for getting VNC running - https://cdcvs.fnal.gov/redmine/projects/dune/wiki/Using_VNC_Connections_on_the_dunegpvms

Execute the following commands after logging in to the DUNE interactive node, creating the directories above and executing the Simulation and Reconstruction commands -

# run the following command to test your connection.  If nothing appears on your vnc display, the setup is not correct
xlogo

# run the event display.  You will have to click through the TPC selector button to find your particle:
lar -c evd_dune10kt.fcl prod_muminus_0.1-5.0GeV_isotropic_dune10kt_1x2x6_gen_g4_detsim_reco.root

Look at the FIFE Monitoring

The FIFE Monitoring displays statistics on jobs, storage uses etc. This can be a useful place to look if things are not happening as you expect -

https://fifemon.fnal.gov/monitor/dashboard/db/experiment-overview?var-experiment=dune

You will be prompted to log in with your Fermilab services username and password.
Click on the graphs to see what you can do.

Extra Credit: Looking Around

Execute the following commands after logging in to the DUNE interactive node and running the setup commands above -

# There are four colon-separated search paths for finding stuff. They are built with the setup command above
# For commands:
echo $PATH

# for shared libraries:
echo $LD_LIBRARY_PATH

# for fcl files (which steer a job)
echo $FHICL_FILE_PATH

# for other files that the framework needs (like GDML files for geometry, or photon libraries):
echo $FW_SEARCH_PATH

# ups sets up environment variables per package that point to the locations of the installed products
echo $DUNETPC_DIR

# subdirectories in there include source code, headers, pre-built and installed libraries, and fcl files

# look at them all!
env | grep DIR

Example: Building a New Analysis Module Based on larexamples

We will be coaching users through this during the afternoon session at the tutorial. Follow the link below for instructions.

Analysis_Module_From_Example_August2017