Using NOvASoft on the GPVM nodes » History » Version 14

Craig Group, 09/13/2012 11:55 AM

1 1 Brian Rebel
2 1 Brian Rebel
3 11 Andrew Norman
h1. Using NOvASoft on the NOvA Interactive nodes
4 1 Brian Rebel
5 1 Brian Rebel
h2. Available nodes
6 1 Brian Rebel
7 11 Andrew Norman
NOvA has a small pool of computing nodes that have been configured with our experiment's software, disk resources and user accounts.  The nodes are part of the "General Purpose Computing Facilities" (GPCF) at Fermilab and can be accessed both from Fermilab as well as offsite.  
8 1 Brian Rebel
9 11 Andrew Norman
Other Intensity Frontier (IF) experiments have similar pools of computing resources and can be accessed in a similar manner to that described here for NOvA.
10 11 Andrew Norman
11 11 Andrew Norman
To login to the NOvA interactive nodes, you login to "".  This name performs some limited load balancing between all the computers to ensure that not everyone ends up on the same machine.  In general you will will be able to log in using a command like:
12 11 Andrew Norman
13 1 Brian Rebel
14 11 Andrew Norman
15 1 Brian Rebel
16 1 Brian Rebel
17 11 Andrew Norman
You will be logged into a machine with a name like "" (i.e. novagpvm01, novagpvm02, etc...) and from there you will be able to access all of the standard NOvA offline resources.
18 11 Andrew Norman
19 11 Andrew Norman
If you need to bypass the load balancing system (the load balancing is designed to help everyone be a good neighbor, so think twice before bypassing it) you can directly log into any one of the machines in the NOvA offline cluster by going directly to it via its fully qualified name (i.e. or it's short form name (i.e.  This is most useful if for some reason you managed to leave something running on a machine and need to go back and check on it (Note:  Don't leave long CPU intensive things running on the interactive nodes!)
20 11 Andrew Norman
21 14 Craig Group
Currently there are 10 interactive nodes (as of 20JUL2012):
22 11 Andrew Norman
23 11 Andrew Norman
24 11 Andrew Norman
25 11 Andrew Norman
26 11 Andrew Norman
27 11 Andrew Norman
28 11 Andrew Norman
29 11 Andrew Norman
30 11 Andrew Norman
31 11 Andrew Norman
32 11 Andrew Norman
33 11 Andrew Norman
Any one of these nodes will give you access to both the FermiGrid and local batch clusters.
34 11 Andrew Norman
35 6 Mayly Sanchez
If you have just received notification that you have an account on one of these machines, your login shell is likely bash.  If you prefer a different login shell you need to submit a ServiceDesk ticket to get it changed.
36 1 Brian Rebel
37 6 Mayly Sanchez
h2. Kerberos
38 8 Gavin Davies
39 10 Jose Sepulveda-Quiroz
Users must have a valid kerberos ticket to access Fermilab computing at the time an attempt to log into a Fermilab machine. The ticket is obtained by executing the following command at a terminal prompt:
40 8 Gavin Davies
41 8 Gavin Davies
$ kinit principal@FNAL.GOV
42 8 Gavin Davies
43 1 Brian Rebel
where principal is the user's kerberos principal. If a user is attempting to access the repository from a non-Fermilab machine, the following lines must be in the users' .ssh/config:
44 6 Mayly Sanchez
45 1 Brian Rebel
Host *
46 6 Mayly Sanchez
ForwardAgent yes
47 6 Mayly Sanchez
ForwardX11 yes
48 6 Mayly Sanchez
ForwardX11Trusted yes
49 6 Mayly Sanchez
GSSAPIAuthentication yes
50 6 Mayly Sanchez
GSSAPIDelegateCredentials yes
51 11 Andrew Norman
GSSAPITrustDns yes
52 12 Gavin Davies
GSSAPIKeyExchange yes
53 12 Gavin Davies
54 12 Gavin Davies
55 12 Gavin Davies
You may also need to add the following in the case of connection issues:
56 12 Gavin Davies
57 12 Gavin Davies
StrictHostKeyChecking no
58 12 Gavin Davies
59 6 Mayly Sanchez
60 11 Andrew Norman
61 6 Mayly Sanchez
In case of trouble when connecting via ssh (permission denied error) the reason can be in the OpenSSH client, the following client is compatible with Fermilab Kerberos authentification:
62 13 Gavin Davies
OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
63 13 Gavin Davies
64 13 Gavin Davies
Some users have experienced problems using the option "GSSAPIKeyExchange yes".
65 13 Gavin Davies
~/.ssh/config: line 8: Bad configuration option: GSSAPIKeyExchange
66 13 Gavin Davies
~/.ssh/config: terminating, 1 bad configuration options
67 13 Gavin Davies
This problem goes away if this option is removed from their .ssh/config 
68 13 Gavin Davies
69 1 Brian Rebel
70 7 Mayly Sanchez
h2. Setting up NOvASoft
71 8 Gavin Davies
72 9 Gavin Davies
There is a setup script provided to get the environment ready for using NOvASoft on these machines.  To use it, one does (for tags S11.04.30 and later including development): 
73 8 Gavin Davies
74 7 Mayly Sanchez
$ source /grid/fermiapp/nova/novaart/novasoft/setup/setup_novasoft_nusoft.(c)sh
75 7 Mayly Sanchez
76 1 Brian Rebel
If you are using tagged release S11.04.09 and earlier do: 
77 3 Brian Rebel
78 7 Mayly Sanchez
@$source /grid/fermiapp/nova/novaart/novasoft/releases/development/setup/setup_novasoft_ifcluster.(c)sh@
79 1 Brian Rebel
80 2 Brian Rebel
The above commands will set your $PATH and $LD_LIBRARY_PATH variables as well as the variables that define the locations of the necessary external packages.
81 1 Brian Rebel
82 1 Brian Rebel
While the public release of the code is located in /grid/fermiapp/nova/novaart/novasoft/releases/development/, the average user should _never_ make any files in that directory.  It has limited space and is only for code releases.  Instead, use the disk space described in the next section.
83 1 Brian Rebel
84 1 Brian Rebel
h2. Disk Space
85 8 Gavin Davies
86 8 Gavin Davies
The user space for those logging into these nodes is 
87 8 Gavin Davies
88 1 Brian Rebel
89 1 Brian Rebel
90 1 Brian Rebel
This is where users should store their test releases as well as any analysis files.
91 8 Gavin Davies
92 8 Gavin Davies
Any data or Monte Carlo files for general consumption by the experiment should be stored in 
93 8 Gavin Davies
94 1 Brian Rebel
95 1 Brian Rebel
96 1 Brian Rebel
When operating on the grid, the /data directories cannot have executables run from them, executables can only be run from the /app directories.