Using NOvASoft on the GPVM nodes » History » Version 25
Jan Zirnstein, 04/03/2014 04:34 PM
Don't confuse people with short names that are only established for half the GPVMs anyway. Removing old setup instructions. Add information about the maxopt build.
1 | 1 | Brian Rebel | {{toc}} |
---|---|---|---|
2 | 1 | Brian Rebel | |
3 | 11 | Andrew Norman | h1. Using NOvASoft on the NOvA Interactive nodes |
4 | 1 | Brian Rebel | |
5 | 1 | Brian Rebel | h2. Available nodes |
6 | 1 | Brian Rebel | |
7 | 11 | Andrew Norman | NOvA has a small pool of computing nodes that have been configured with our experiment's software, disk resources and user accounts. The nodes are part of the "General Purpose Computing Facilities" (GPCF) at Fermilab and can be accessed both from Fermilab as well as offsite. |
8 | 1 | Brian Rebel | |
9 | 11 | Andrew Norman | Other Intensity Frontier (IF) experiments have similar pools of computing resources and can be accessed in a similar manner to that described here for NOvA. |
10 | 11 | Andrew Norman | |
11 | 11 | Andrew Norman | To login to the NOvA interactive nodes, you login to "nova-offline.fnal.gov". This name performs some limited load balancing between all the computers to ensure that not everyone ends up on the same machine. In general you will will be able to log in using a command like: |
12 | 11 | Andrew Norman | |
13 | 1 | Brian Rebel | <pre> |
14 | 11 | Andrew Norman | ssh myusername@nova-offline.fnal.gov |
15 | 1 | Brian Rebel | </pre> |
16 | 1 | Brian Rebel | |
17 | 11 | Andrew Norman | You will be logged into a machine with a name like "novagpvmXX.fnal.gov" (i.e. novagpvm01, novagpvm02, etc...) and from there you will be able to access all of the standard NOvA offline resources. |
18 | 11 | Andrew Norman | |
19 | 18 | Arthur Kreymer | Load balancing is designed to help everyone be a good neighbor, so think twice before bypassing it. |
20 | 18 | Arthur Kreymer | Load balancing does not work for all clients, see the Kerberos notes below. |
21 | 25 | Jan Zirnstein | If you need to bypass the load balancing system you can directly log into any one of the machines in the NOvA offline cluster via its full name (i.e. novagpvm01.fnal.gov). |
22 | 18 | Arthur Kreymer | This is most useful if for some reason you managed to leave something running on a machine and need to go back and check on it |
23 | 18 | Arthur Kreymer | (Note: Don't leave long CPU intensive things running on the interactive nodes!) |
24 | 1 | Brian Rebel | |
25 | 18 | Arthur Kreymer | |
26 | 18 | Arthur Kreymer | |
27 | 14 | Craig Group | Currently there are 10 interactive nodes (as of 20JUL2012): |
28 | 11 | Andrew Norman | |
29 | 11 | Andrew Norman | <pre> |
30 | 11 | Andrew Norman | *novagpvm01.fnal.gov |
31 | 11 | Andrew Norman | *novagpvm02.fnal.gov |
32 | 11 | Andrew Norman | *novagpvm03.fnal.gov |
33 | 11 | Andrew Norman | *novagpvm04.fnal.gov |
34 | 11 | Andrew Norman | *novagpvm05.fnal.gov |
35 | 11 | Andrew Norman | *.... |
36 | 11 | Andrew Norman | *novagpvm10.fnal.gov |
37 | 11 | Andrew Norman | </pre> |
38 | 11 | Andrew Norman | |
39 | 11 | Andrew Norman | Any one of these nodes will give you access to both the FermiGrid and local batch clusters. |
40 | 11 | Andrew Norman | |
41 | 6 | Mayly Sanchez | If you have just received notification that you have an account on one of these machines, your login shell is likely bash. If you prefer a different login shell you need to submit a ServiceDesk ticket to get it changed. |
42 | 1 | Brian Rebel | |
43 | 6 | Mayly Sanchez | h2. Kerberos |
44 | 8 | Gavin Davies | |
45 | 10 | Jose Sepulveda-Quiroz | Users must have a valid kerberos ticket to access Fermilab computing at the time an attempt to log into a Fermilab machine. The ticket is obtained by executing the following command at a terminal prompt: |
46 | 8 | Gavin Davies | <pre> |
47 | 8 | Gavin Davies | $ kinit principal@FNAL.GOV |
48 | 8 | Gavin Davies | </pre> |
49 | 1 | Brian Rebel | where principal is the user's kerberos principal. If a user is attempting to access the repository from a non-Fermilab machine, the following lines must be in the users' .ssh/config: |
50 | 6 | Mayly Sanchez | <pre> |
51 | 1 | Brian Rebel | Host *.fnal.gov |
52 | 6 | Mayly Sanchez | ForwardAgent yes |
53 | 6 | Mayly Sanchez | ForwardX11 yes |
54 | 6 | Mayly Sanchez | ForwardX11Trusted yes |
55 | 24 | Rick Tesarek | # GSSAPIAuthentication yes |
56 | 24 | Rick Tesarek | # GSSAPIDelegateCredentials yes |
57 | 1 | Brian Rebel | GSSAPITrustDns yes |
58 | 24 | Rick Tesarek | # GSSAPIKeyExchange yes |
59 | 24 | Rick Tesarek | # ServerAliveInterval 60 |
60 | 20 | Himansu Sahoo | </pre> |
61 | 24 | Rick Tesarek | 8/2/13 - RJT - I found that I needed to comment out the lines as shown above. |
62 | 12 | Gavin Davies | |
63 | 12 | Gavin Davies | You may also need to add the following in the case of connection issues: |
64 | 12 | Gavin Davies | <pre> |
65 | 12 | Gavin Davies | StrictHostKeyChecking no |
66 | 12 | Gavin Davies | UserKnownHostsFile=/dev/null |
67 | 6 | Mayly Sanchez | </pre> |
68 | 22 | Biao Wang | Some new users may find out that the krb5.conf post online does not work on their Mac OSX 10.8 system, in that case, you may need to ask a current mac researcher for his krb5.conf on his machine. |
69 | 6 | Mayly Sanchez | In case of trouble when connecting via ssh (permission denied error) the reason can be in the OpenSSH client, the following client is compatible with Fermilab Kerberos authentification: |
70 | 13 | Gavin Davies | OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008 |
71 | 13 | Gavin Davies | |
72 | 13 | Gavin Davies | Some users have experienced problems using the option "GSSAPIKeyExchange yes". |
73 | 13 | Gavin Davies | ~/.ssh/config: line 8: Bad configuration option: GSSAPIKeyExchange |
74 | 13 | Gavin Davies | ~/.ssh/config: terminating, 1 bad configuration options |
75 | 13 | Gavin Davies | This problem goes away if this option is removed from their .ssh/config |
76 | 1 | Brian Rebel | |
77 | 18 | Arthur Kreymer | Some Windows/PUTTY users are unable to connect to the nova-offline load balancing address, and must connect directly to particular nodes. |
78 | 1 | Brian Rebel | |
79 | 7 | Mayly Sanchez | h2. Setting up NOvASoft |
80 | 8 | Gavin Davies | |
81 | 17 | Andrew Norman | When using the nova-offline computing cluster at Fermilab, the full suite of NOvA software has already been installed and properly configured. To use the software there are a set of setup scripts that you should use to configure your session. |
82 | 17 | Andrew Norman | |
83 | 17 | Andrew Norman | There are a couple of methods to use these setup scripts. The recommended method is to put the following code snippet into your login files (your .bashrc or .bash_profile if you are using bash as your shell): |
84 | 17 | Andrew Norman | |
85 | 17 | Andrew Norman | <pre> |
86 | 23 | Rick Tesarek | function setup_novaoffline() { |
87 | 19 | Jonathan Paley | source /grid/fermiapp/nova/novaart/novasvn/srt/srt.sh |
88 | 17 | Andrew Norman | export EXTERNALS=/nusoft/app/externals |
89 | 17 | Andrew Norman | source $SRT_DIST/setup/setup_novasoft.sh "$@" |
90 | 17 | Andrew Norman | } |
91 | 17 | Andrew Norman | </pre> |
92 | 17 | Andrew Norman | |
93 | 17 | Andrew Norman | Then to setup the software you can login and type: |
94 | 17 | Andrew Norman | |
95 | 17 | Andrew Norman | <pre> |
96 | 17 | Andrew Norman | setup_novaoffline |
97 | 17 | Andrew Norman | </pre> |
98 | 17 | Andrew Norman | |
99 | 25 | Jan Zirnstein | If you want to setup a specific release you will append the tag name [[History of Tagged Releases]] to the setup command: |
100 | 17 | Andrew Norman | <pre> |
101 | 17 | Andrew Norman | setup_novaoffline -r <tag-release> |
102 | 17 | Andrew Norman | </pre> |
103 | 1 | Brian Rebel | |
104 | 25 | Jan Zirnstein | When you are done developing a piece of software and you are ready to run it, consider setting up novasoft in maxopt mode. This sets certain compiler flags, and uses the maxopt version of precompiled externals, to optimize the execution of your code. Read: It will run at least twice as fast. Do enable this build option you would do: |
105 | 19 | Jonathan Paley | <pre> |
106 | 25 | Jan Zirnstein | setup_novaoffline -b maxopt |
107 | 15 | Gavin Davies | </pre> |
108 | 1 | Brian Rebel | |
109 | 2 | Brian Rebel | The above commands will set your $PATH and $LD_LIBRARY_PATH variables as well as the variables that define the locations of the necessary external packages. |
110 | 1 | Brian Rebel | |
111 | 19 | Jonathan Paley | While the public release of the code is located in /grid/fermiapp/nova/novaart/novasvn/releases/development/, the average user should _never_ make any files in that directory. It has limited space and is only for code releases. Instead, use the disk space described in the next section. |
112 | 1 | Brian Rebel | |
113 | 1 | Brian Rebel | h2. Disk Space |
114 | 8 | Gavin Davies | |
115 | 8 | Gavin Davies | The user space for those logging into these nodes is |
116 | 8 | Gavin Davies | <pre> |
117 | 1 | Brian Rebel | /nova/app/users |
118 | 1 | Brian Rebel | </pre> |
119 | 1 | Brian Rebel | This is where users should store their test releases as well as any analysis files. |
120 | 8 | Gavin Davies | |
121 | 21 | Andrew Norman | There is a special location reserved for files that are generated by the different analysis groups. This area is: |
122 | 1 | Brian Rebel | <pre> |
123 | 21 | Andrew Norman | /nova/ana |
124 | 21 | Andrew Norman | </pre> |
125 | 21 | Andrew Norman | And has subdirectories for each of the analysis or working groups. |
126 | 21 | Andrew Norman | |
127 | 21 | Andrew Norman | In addtion, the production group maintains the large shared disk that stores our raw data, processed data and Monte Carlo samples that are used for general consumption by the experiments. You should use this area only if directed by your analysis/production coordinator (since filling this area can cause our production and reprocessing projects to fail). |
128 | 21 | Andrew Norman | |
129 | 21 | Andrew Norman | This area is: |
130 | 21 | Andrew Norman | <pre> |
131 | 1 | Brian Rebel | /nova/data |
132 | 1 | Brian Rebel | </pre> |
133 | 21 | Andrew Norman | |
134 | 21 | Andrew Norman | When operating on the grid, the /nova/data and /nova/ana directories cannot have executables run from them, executables can only be run from the /nova/app directories. |