Admin guide » History » Version 16
Herbert Greenlee, 01/06/2015 04:26 PM
1 | 8 | Herbert Greenlee | {{toc}} |
---|---|---|---|
2 | 8 | Herbert Greenlee | |
3 | 1 | Herbert Greenlee | h1. The experiment environment setup script |
4 | 1 | Herbert Greenlee | |
5 | 12 | Herbert Greenlee | Every experiment has a script for initializing its standard software environment, which users source from a standard place. This script may do any or all of the following things. |
6 | 2 | Herbert Greenlee | |
7 | 2 | Herbert Greenlee | * Initialize ups products system and products areas. |
8 | 2 | Herbert Greenlee | * Set up some basic ups products. |
9 | 2 | Herbert Greenlee | * Set environment variables. |
10 | 2 | Herbert Greenlee | |
11 | 13 | Herbert Greenlee | This section explains the recommended way of writing this script. |
12 | 2 | Herbert Greenlee | |
13 | 2 | Herbert Greenlee | h2. Initializing ups |
14 | 2 | Herbert Greenlee | |
15 | 2 | Herbert Greenlee | After initializing ups, the following should be the case: |
16 | 2 | Herbert Greenlee | |
17 | 2 | Herbert Greenlee | * The executable @ups@ should be on the execution path. |
18 | 2 | Herbert Greenlee | * Command or alias @setup@ should be defined. |
19 | 3 | Herbert Greenlee | * The products path (environment variable @PRODUCTS@) should be defined as a colon-separated list of ups products areas. |
20 | 2 | Herbert Greenlee | |
21 | 4 | Herbert Greenlee | With respect to the last point, the products path may include the following ups products areas (in order). |
22 | 2 | Herbert Greenlee | |
23 | 2 | Herbert Greenlee | * Experiment-specific products (e.g. @/grid/fermiapp/products/uboone@). |
24 | 2 | Herbert Greenlee | * Larsoft products (for larsoft experiments, e.g. @/grid/fermiapp/products/larsoft@). |
25 | 2 | Herbert Greenlee | * Fermilab common products (e.g. @/grid/fermiapp/products/common/db@). |
26 | 2 | Herbert Greenlee | |
27 | 5 | Herbert Greenlee | Generally, ups is initialized by sourcing a script called "@setups@" (or "@setups.sh@" or "@setups.csh@", the name can vary) from some standard place. It is acceptable to source multiple "@setups@" scripts from multiple products areas, as well as extending the products path to include additional products areas manually. |
28 | 5 | Herbert Greenlee | |
29 | 5 | Herbert Greenlee | For larsoft experiments on nodes where the @/grid/fermiapp@ filesystem is mounted, the main @setups@ file path is @/grid/fermiapp/products/larsoft/setups@. |
30 | 2 | Herbert Greenlee | |
31 | 6 | Herbert Greenlee | When initializing ups from the oasis cvmfs products area, the main larsoft @setups@ file path is @/cvmfs/oasis.opensciencegrid.org/fermilab/products/larsoft/setups.for.cvmfs@. |
32 | 2 | Herbert Greenlee | |
33 | 9 | Herbert Greenlee | h2. Ups products setup |
34 | 9 | Herbert Greenlee | |
35 | 9 | Herbert Greenlee | The experiment setup script may set up some basic ups products. Any ups product that is setup in the experiment setup script should be setup as a "current" version. Any product that requires a numbered version or qualifier in the setup command should not be setup in the experiment setup script. Microboone's experiment script sets up the following products that are used by the build environment. |
36 | 9 | Herbert Greenlee | |
37 | 9 | Herbert Greenlee | * git |
38 | 9 | Herbert Greenlee | * gitflow |
39 | 9 | Herbert Greenlee | * mrb |
40 | 9 | Herbert Greenlee | |
41 | 9 | Herbert Greenlee | h2. Environment variables |
42 | 9 | Herbert Greenlee | |
43 | 10 | Herbert Greenlee | Here are some environment variables that must or can be set. |
44 | 10 | Herbert Greenlee | |
45 | 10 | Herbert Greenlee | * MRB_PROJECT - should be set as "larsoft" for larsoft experiments that use mrb. |
46 | 10 | Herbert Greenlee | * FW_SEARCH_PATH - Art framework file search path. |
47 | 10 | Herbert Greenlee | |
48 | 10 | Herbert Greenlee | The standard experiment name may be defined in several different environment variables. |
49 | 10 | Herbert Greenlee | |
50 | 10 | Herbert Greenlee | * EXPERIMENT - Used by @ifdh@. |
51 | 10 | Herbert Greenlee | * SAM_EXPERIMENT - Used by @sam@. |
52 | 10 | Herbert Greenlee | * JOBSUB_GROUP - Used by @jobsub_client@. |
53 | 9 | Herbert Greenlee | |
54 | 9 | Herbert Greenlee | |
55 | 2 | Herbert Greenlee | |
56 | 2 | Herbert Greenlee | |
57 | 1 | Herbert Greenlee | h1. Experiment-specific hooks in @project.py@. |
58 | 13 | Herbert Greenlee | |
59 | 13 | Herbert Greenlee | @Project.py@ is an executable python script. Some functionality of @project.py@ is implemented in a python module called @project_utilities@, which module can be found in subdirectory @python@ of ups product @larbatch@. Module @project_utilities@ is on the python path when you setup @larbatch@. The last statement of @project_utilities.py@ is |
60 | 13 | Herbert Greenlee | |
61 | 13 | Herbert Greenlee | <pre> |
62 | 13 | Herbert Greenlee | from experiment_utilities import * |
63 | 13 | Herbert Greenlee | </pre> |
64 | 13 | Herbert Greenlee | |
65 | 14 | Herbert Greenlee | This statement allows @experiment_utilities@ to override any object or function defined in @project_utilities@. The intention is that each experiment will have its own version of @experiment_utilities@ available on the python path. |
66 | 13 | Herbert Greenlee | |
67 | 13 | Herbert Greenlee | There are several functions defined in @project_utilities@ that are candidates for being overridden by @experiment_utilities@. Some functions must be overridden for corresponding functionality to be available. Some functions have a default implementation that can be optionally overridden. |
68 | 1 | Herbert Greenlee | |
69 | 14 | Herbert Greenlee | Here are some of the functions that can or must be overriden: |
70 | 14 | Herbert Greenlee | |
71 | 15 | Herbert Greenlee | * @get_experiment()@ |
72 | 16 | Herbert Greenlee | Returns the standard experiment name. The default implementation in @project_utilities@, which should be acceptable in many cases, uses environment variables to determine the experiment name. |
73 | 15 | Herbert Greenlee | |
74 | 16 | Herbert Greenlee | * @get_setup_script()@ |
75 | 1 | Herbert Greenlee | Returns the full path of the experiment initialization script on the current node. The default implementation in @project_utilities@ raises an exception. |
76 | 1 | Herbert Greenlee | |
77 | 16 | Herbert Greenlee | * @get_bluearc_server()@ |
78 | 16 | Herbert Greenlee | Returns the fictitious sam bluearc server nodename (used for specifying sam locations of files on bluearc). The default implementation in @project_utilities@ returns the string containing the experiment name + 'data:'. |
79 | 16 | Herbert Greenlee | |
80 | 16 | Herbert Greenlee | * @get_dcache_server()@ |
81 | 16 | Herbert Greenlee | Returns the fictitious sam dCache server nodename (used for specifying sam locations of files on dCache). The default implementation in @project_utilities@ returns the string 'fnal-dcache:'. |
82 | 16 | Herbert Greenlee | |
83 | 1 | Herbert Greenlee | * @get_dropbox(filename)@ |
84 | 16 | Herbert Greenlee | Returns the full path of the File Transfer Service dropbox corresponding to a file that has metadata in sam. The default implementation in @project_utilities@ raises an exception. |
85 | 16 | Herbert Greenlee | |
86 | 16 | Herbert Greenlee | * @get_role()@ |
87 | 16 | Herbert Greenlee | Returns a string containing voms role. This function has a default implementation in @project_utilities@ that will usually return 'Analysis' or 'Production' depending on the user name. |
88 | 1 | Herbert Greenlee | |
89 | 15 | Herbert Greenlee | * @get_sam_metadata(project, stage)@ |
90 | 15 | Herbert Greenlee | Returns a string containing an fcl fragment for generating sam metadata. The default implementation in @project_utilities@ returns an empty string. The arguments are python class objects of types ProjectDef and StageDef (defined in larbatch/python). |
91 | 15 | Herbert Greenlee | |
92 | 15 | Herbert Greenlee | * @get_dimensions(project, stage)@ |
93 | 15 | Herbert Greenlee | Returns a string containing a sam query predicate for querying files generated by the specified project and stage. The default implementation in @project_utilities@ raises an exception. The arguments are python class objects of types ProjectDef and StageDef (defined in larbatch/python). |