Project

General

Profile

Using NOvASoft on the GPVM nodes » History » Version 24

Rick Tesarek, 08/02/2013 03:46 PM
Fix .ssh/config file

1 1 Brian Rebel
{{toc}}
2 1 Brian Rebel
3 11 Andrew Norman
h1. Using NOvASoft on the NOvA Interactive nodes
4 1 Brian Rebel
5 1 Brian Rebel
h2. Available nodes
6 1 Brian Rebel
7 11 Andrew Norman
NOvA has a small pool of computing nodes that have been configured with our experiment's software, disk resources and user accounts.  The nodes are part of the "General Purpose Computing Facilities" (GPCF) at Fermilab and can be accessed both from Fermilab as well as offsite.  
8 1 Brian Rebel
9 11 Andrew Norman
Other Intensity Frontier (IF) experiments have similar pools of computing resources and can be accessed in a similar manner to that described here for NOvA.
10 11 Andrew Norman
11 11 Andrew Norman
To login to the NOvA interactive nodes, you login to "nova-offline.fnal.gov".  This name performs some limited load balancing between all the computers to ensure that not everyone ends up on the same machine.  In general you will will be able to log in using a command like:
12 11 Andrew Norman
13 1 Brian Rebel
<pre>
14 11 Andrew Norman
ssh myusername@nova-offline.fnal.gov
15 1 Brian Rebel
</pre>
16 1 Brian Rebel
17 11 Andrew Norman
You will be logged into a machine with a name like "novagpvmXX.fnal.gov" (i.e. novagpvm01, novagpvm02, etc...) and from there you will be able to access all of the standard NOvA offline resources.
18 11 Andrew Norman
19 18 Arthur Kreymer
Load balancing is designed to help everyone be a good neighbor, so think twice before bypassing it.
20 18 Arthur Kreymer
Load balancing does not work for all clients, see the Kerberos notes below.
21 18 Arthur Kreymer
If you need to bypass the load balancing system  you can directly log into any one of the machines in the NOvA offline cluster via its full name (i.e. novagpvm01.fnal.gov) or its short form name (i.e. nova01.fnal.gov).  
22 18 Arthur Kreymer
This is most useful if for some reason you managed to leave something running on a machine and need to go back and check on it 
23 18 Arthur Kreymer
(Note:  Don't leave long CPU intensive things running on the interactive nodes!)
24 1 Brian Rebel
25 18 Arthur Kreymer
26 18 Arthur Kreymer
27 14 Craig Group
Currently there are 10 interactive nodes (as of 20JUL2012):
28 11 Andrew Norman
29 11 Andrew Norman
<pre>
30 11 Andrew Norman
*novagpvm01.fnal.gov
31 11 Andrew Norman
*novagpvm02.fnal.gov
32 11 Andrew Norman
*novagpvm03.fnal.gov
33 11 Andrew Norman
*novagpvm04.fnal.gov
34 11 Andrew Norman
*novagpvm05.fnal.gov
35 11 Andrew Norman
*....
36 11 Andrew Norman
*novagpvm10.fnal.gov
37 11 Andrew Norman
</pre>
38 11 Andrew Norman
39 11 Andrew Norman
Any one of these nodes will give you access to both the FermiGrid and local batch clusters.
40 11 Andrew Norman
41 6 Mayly Sanchez
If you have just received notification that you have an account on one of these machines, your login shell is likely bash.  If you prefer a different login shell you need to submit a ServiceDesk ticket to get it changed.
42 1 Brian Rebel
43 6 Mayly Sanchez
h2. Kerberos
44 8 Gavin Davies
45 10 Jose Sepulveda-Quiroz
Users must have a valid kerberos ticket to access Fermilab computing at the time an attempt to log into a Fermilab machine. The ticket is obtained by executing the following command at a terminal prompt:
46 8 Gavin Davies
<pre>
47 8 Gavin Davies
$ kinit principal@FNAL.GOV
48 8 Gavin Davies
</pre>
49 1 Brian Rebel
where principal is the user's kerberos principal. If a user is attempting to access the repository from a non-Fermilab machine, the following lines must be in the users' .ssh/config:
50 6 Mayly Sanchez
<pre>
51 1 Brian Rebel
Host *.fnal.gov
52 6 Mayly Sanchez
ForwardAgent yes
53 6 Mayly Sanchez
ForwardX11 yes
54 6 Mayly Sanchez
ForwardX11Trusted yes
55 24 Rick Tesarek
#  GSSAPIAuthentication yes
56 24 Rick Tesarek
#  GSSAPIDelegateCredentials yes
57 1 Brian Rebel
GSSAPITrustDns yes
58 24 Rick Tesarek
#  GSSAPIKeyExchange yes
59 24 Rick Tesarek
#  ServerAliveInterval 60
60 20 Himansu Sahoo
</pre>
61 24 Rick Tesarek
8/2/13 - RJT - I found that I needed to comment out the lines as shown above.
62 12 Gavin Davies
63 12 Gavin Davies
You may also need to add the following in the case of connection issues:
64 12 Gavin Davies
<pre>
65 12 Gavin Davies
StrictHostKeyChecking no
66 12 Gavin Davies
UserKnownHostsFile=/dev/null
67 6 Mayly Sanchez
</pre>
68 22 Biao Wang
Some new users may find out that the krb5.conf post online does not work on their Mac OSX 10.8 system, in that case, you may need to ask a current mac researcher for his krb5.conf on his machine.
69 6 Mayly Sanchez
In case of trouble when connecting via ssh (permission denied error) the reason can be in the OpenSSH client, the following client is compatible with Fermilab Kerberos authentification:
70 13 Gavin Davies
OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
71 13 Gavin Davies
72 13 Gavin Davies
Some users have experienced problems using the option "GSSAPIKeyExchange yes".
73 13 Gavin Davies
~/.ssh/config: line 8: Bad configuration option: GSSAPIKeyExchange
74 13 Gavin Davies
~/.ssh/config: terminating, 1 bad configuration options
75 13 Gavin Davies
This problem goes away if this option is removed from their .ssh/config 
76 1 Brian Rebel
77 18 Arthur Kreymer
Some Windows/PUTTY users are unable to connect to the nova-offline load balancing address, and must connect directly to particular nodes.
78 1 Brian Rebel
79 7 Mayly Sanchez
h2. Setting up NOvASoft
80 8 Gavin Davies
81 17 Andrew Norman
When using the nova-offline computing cluster at Fermilab, the full suite of NOvA software has already been installed and properly configured.  To use the software there are a set of setup scripts that you should use to configure your session.
82 17 Andrew Norman
83 17 Andrew Norman
There are a couple of methods to use these setup scripts.  The recommended method is to put the following code snippet into your login files (your .bashrc or .bash_profile if you are using bash as your shell):
84 17 Andrew Norman
85 17 Andrew Norman
<pre>
86 23 Rick Tesarek
function setup_novaoffline() {
87 19 Jonathan Paley
  source /grid/fermiapp/nova/novaart/novasvn/srt/srt.sh
88 17 Andrew Norman
  export EXTERNALS=/nusoft/app/externals
89 17 Andrew Norman
  source $SRT_DIST/setup/setup_novasoft.sh "$@"
90 17 Andrew Norman
}
91 17 Andrew Norman
</pre>
92 17 Andrew Norman
93 17 Andrew Norman
Then to setup the software you can login and type:
94 17 Andrew Norman
95 17 Andrew Norman
<pre>
96 17 Andrew Norman
setup_novaoffline
97 17 Andrew Norman
</pre>
98 17 Andrew Norman
99 17 Andrew Norman
If you want to setup a specific release you will append the tag name to the setup command:
100 17 Andrew Norman
<pre>
101 17 Andrew Norman
setup_novaoffline -r <tag-release>
102 17 Andrew Norman
</pre>
103 17 Andrew Norman
104 17 Andrew Norman
105 17 Andrew Norman
106 17 Andrew Norman
OLDER Instructions:
107 15 Gavin Davies
108 15 Gavin Davies
To source the 'development' do 
109 1 Brian Rebel
<pre>
110 19 Jonathan Paley
% source /grid/fermiapp/nova/novaart/novasvn/srt/srt.sh
111 15 Gavin Davies
% export EXTERNALS=/nusoft/app/externals
112 16 Arthur Kreymer
% source $SRT_DIST/setup/setup_novasoft.sh
113 1 Brian Rebel
</pre>
114 1 Brian Rebel
115 15 Gavin Davies
To source a 'tagged release' do
116 15 Gavin Davies
<pre>
117 19 Jonathan Paley
% source /grid/fermiapp/nova/novaart/novasvn/srt/srt.sh
118 15 Gavin Davies
% export EXTERNALS=/nusoft/app/externals
119 19 Jonathan Paley
% source /grid/fermiapp/nova/novaart/novasvn/setup/setup_novasoft.sh -r <tag-release>
120 15 Gavin Davies
</pre>
121 15 Gavin Davies
where '<tag-release>' is the label for the tagged release.
122 1 Brian Rebel
123 2 Brian Rebel
The above commands will set your $PATH and $LD_LIBRARY_PATH variables as well as the variables that define the locations of the necessary external packages.
124 1 Brian Rebel
125 19 Jonathan Paley
While the public release of the code is located in /grid/fermiapp/nova/novaart/novasvn/releases/development/, the average user should _never_ make any files in that directory.  It has limited space and is only for code releases.  Instead, use the disk space described in the next section.
126 1 Brian Rebel
127 1 Brian Rebel
h2. Disk Space
128 8 Gavin Davies
129 8 Gavin Davies
The user space for those logging into these nodes is 
130 8 Gavin Davies
<pre>
131 1 Brian Rebel
/nova/app/users
132 1 Brian Rebel
</pre>
133 1 Brian Rebel
This is where users should store their test releases as well as any analysis files.
134 8 Gavin Davies
135 21 Andrew Norman
There is a special location reserved for files that are generated by the different analysis groups.  This area is:
136 1 Brian Rebel
<pre>
137 21 Andrew Norman
/nova/ana
138 21 Andrew Norman
</pre>
139 21 Andrew Norman
And has subdirectories for each of the analysis or working groups.
140 21 Andrew Norman
141 21 Andrew Norman
In addtion, the production group maintains the large shared disk that stores our raw data, processed data and Monte Carlo samples that are used for general consumption by the experiments.  You should use this area only if directed by your analysis/production coordinator (since filling this area can cause our production and reprocessing projects to fail).
142 21 Andrew Norman
143 21 Andrew Norman
This area is:
144 21 Andrew Norman
<pre>
145 1 Brian Rebel
/nova/data
146 1 Brian Rebel
</pre>
147 21 Andrew Norman
148 21 Andrew Norman
When operating on the grid, the /nova/data and /nova/ana directories cannot have executables run from them, executables can only be run from the /nova/app directories.