 |
 |
 |
 |
Complete Guide and Reference Manual for UPS and UPD | |||
Chapter Contents¶
Chapter 1: Quick! I Just Want to ...
1.1 Why UPS/UPD is Needed
1.2 Install a Product using tar
1.3 Install a Product from KITS; No UPS/UPD on Target Machine
1.4 Install UPS/UPD in a Preset Configuration
1.5 Something is Wrong
1.6 Checking Layout Before ups V5_1
Chapter 1: Quick! I Just Want to ...¶
This chapter is targeted to people who don't know anything about Unix Product Support and Unix Product Distribution UPS/UPD but want to get started quickly without having to read very much.
If you already know what to install, and where, you can skip to section 1.2 Install a Product using tar
1.1 Why UPS/UPD is Needed¶
The software for experiments rely on external tools. These packages and tools are referred to as external products, or simply products. The following is just a partial list, and may not apply to all experiments. It is given to illustrate the types of packages and tools that are external products.
- art -- the event processing framework
- CLHEP -- Class Library for High Energy Physics
- Geant4 -- toolkit for the simulation of the passage of particles through matter.
- git -- a source code management system
- ROOT -- an object oriented framework for large scale data analysis
- UPD -- Unix Product Distribution - used to distribute software products
- UPS -- Unix Product Support -- used to access software products
For Fermilab-based experiments, access to external products is provided by a Fermilab-developed product-management package called Unix Product Support. UPS supports multiple versions of a product and multiple builds per version. For example, a product multiple times for use with different versions of an operating system. A product could be built with a production version of the C++ complier, and with a version under test. It could be built with full compiler optimization or with the maximum debugging features enabled. UPS provides a way to select a particular build. The full identifier of a UPS product includes:
- product name
- version
- flavor
- full set of qualifiers
An important UPS feature, demanded by most experiments as their code evolved, is support for multiple versions of a product and multiple builds (e.g., for different platforms) per version. Another notable feature is the capacity to handle multiple databases of products. On Fermilab computers, login scripts set up the UPS system, providing access to a database of products commonly used at Fermilab.
The act of setting up UPS defines a number of environment variables, one of which is PRODUCTS, a colon-delimited list of directory names. Each directory in PRODUCTS is the name of a UPS database, meaning each directory functions as a repository of information about one or more products. When UPS looks for a product, it checks each directory in PRODUCTS in the order listed, and takes the first match.
To see an example of a UPS repository, the command printenv PRODUCTS (on fermicloud050), gives the directory /fnal/ups/db. The command ls will list the products found there.
$ printenv PRODUCTS /fnal/ups/db $ ls /fnal/ups/db catman fnal_git_notifier lscvs python subversion ups cvs git lxr redmine upd wu_ftpd cvsh ifdhc man ruby upd_libs xxx
For a second example, on novagpvm02 in early 2014, the printenv command gives two directories, with the following products.
printenv PRODUCTS /nusoft/app/externals:/grid/fermiapp/products/common/db % ls /nusoft/app/externals AAAREADME fhiclcpp lhapdf pqxx allinea_tools fluka libsigcpp psycopg2 (and many more) % ls /grid/fermiapp/products/common/db afs ifdhc mako project twisted cherrypy jobsub minos pycurl upd (and many more)
UPS can make multiple variants of a product available to users. This includes different versions, but beyond that, a given version of a product may be built more than one way, e.g., for use by different operating systems (what UPS distinguishes as flavors). Many variants can exist. UPS provides a way to select a particular build via qualifiers. The full identifier of a UPS product includes its product name, its version, its flavor and its full set of qualifiers.
For some UPS products, but not all, the site administrator may define a particular fully-qualified version of the product as the default version. In the language of UPS this notion of default is called the current version. If a current version has been defined for a product, users can set up that product with the command:
$ setup <product>
The UPS system will automatically insert the version and qualifiers of the version that has been declared current. Having a current version is a handy feature since as improvements are added, users automatically get them.
p<>. However the notion of a current version is very dangerous if users want to ensure that software built at one site will build in exactly the same way on all other sites. For this reason, some projects require fully specifying the version number and qualifiers of all products that are required; and in turn, the products make fully qualified requests for the products on which they depend.
Many software products have version numbers that contain dot characters. UPS requires that version numbers not contain any dot characters; by convention, version dots are replaced with underscores. Therefore v0.00.09 becomes v0_00_09. Also by convention, the environment variables are all upper case, regardless of the case used in the product names.
1.2 Install a Product using tar¶
p<>.
The UNIX command tar can be used to combine files into a single file, for easy distribution.
Using tar requires UPS version 5 or later. To use the tar command, the directory structure of the source and destination product areas must be the same. To check this in ups v5_1 or later, run:
% ups layout
If it returns 'unified', then tar can be used. In the following output:
"/scratch/mengel/p": "unified" "/fnal/ups/db": "traditional" "/afs/fnal.gov/ups/db": "traditional"
tar can be used for /scratch/mengel/p.
p<>. For earlier versions of ups , see Checking Layout Before ups V5_1
Once the layout has been verified, the user changes directory to the products area and runs the tar command with the c (create) option.
% cd /products/area % tar czvf /tmp/transfer_me.tgz product1 product2/vx_y_z*
This command packs up all versions of product1 and version vx_y_z of product2 into /tmp/transfer_me.tgz. The tarfile can be transferred however the user wishes to get it to the machine to install it on, such as Google Docs, put it on a webserver, etc. The user changes directory to the products area and runs the tar command with the x (extract) option to get the files.
% cd /other/products/area % tar xzvf /tmp/transfer_me.tgz
Users can also extract only particular packages from the tarfile, by listing exactly what is wanted.
% tar xzvf /tmp/transfer_me.tgz product1
For details on using tar to install a product, please see Chapter 7 .
1.3 Install a Product from KITS; UPS/UPD Exists on Target Machine¶
p<>.
The KITS software repository is available to all computers in the fnal.gov domain and registered users. If you think you don't need to know anything about KITS, be aware that low-level data-handling software is often distributed through UPD, and UPD uses KITS. You may well have scripts that call UPD, which uses KITS.
This section assumes:
- UPS/UPD is installed and configured properly on the target machine, and the defaults are set appropriately
- the product from KITS is to be installed into the standard UPS product area on the machine
- a product tar or zip file made for the OS exists in KITS, and is set as "current" so the version or flavor isn't needed, just the product name
- the product is configured in KITS such that it can be installed in a straightforward manner
On your machine, first setup UPD and verify that the product you want is available from KITS. To do so, issue the following commands:
% setup upd % upd list <product>
You should see output in the format:
DATABASE=/ftp/upsdb
Product=lscvs Version=v1_1 Flavor=NULL
Qualifiers="" Chain=current
If there is no output, or only the DATABASE variable is given, either there's no instance of the product for the OS, or there is one but it's not set to "current". See 24.9 upd list for other options to include.
Assuming the product is there, the next step is to install it. To see where UPS/UPD will install it, first run the command:
% upd install -sv <product>
which will not install it, but will output information about what it would do to install it. Look for the directory it would create for installing the product. Also look for the ups(local) declare... command; the argument to the -z options tells you the database UPS would use to declare the product.
A few notes:
- If UPS/UPD wants to install into AFS space, you can't do a "quick and dirty" install. Check with someone more UPS -savvy and/or AFS-savvy to see if this is the right destination for the product; if so, refer to Chapter 6: Installing Products Using UPD and section 9.3 Installing Products into AFS Space .
- If you want to check if there are databases choices, echo $PRODUCTS to see if there's multiple database paths listed. If so, you can redefine $PRODUCTS such that the one you want is first, then rerun the upd install -sv <product> command to see if the result would change.
- If you want the product in your own area, declared in your own database, again you'll have to know more. See Chapter 6: Installing Products Using UPD and section 12.8.2 Adding a New Database and/or Products Area .
Assuming the upd install -sv command output is what you want, install the product. To do so, enter the command:
% upd install -G "-c" <product>
This installs the product so only setup <product> , is needed before the product is invoked. If you're not sure what command name to use to invoke the product, look for the executable name under $<PRODUCT>_DIR/bin/ .
More information is available in Chapter 6: Installing Products Using UPD . There is a checklist in section 6.4 Checklist for Installing a Product using UPD and examples in section 6.5 Examples .
1.4 Install UPS/UPD in a Preset Configuration¶
This section assumes:
- UPS/UPD is not installed on the target machine, but you want to install it using a simple, preset configuration
- you want to install products from KITS on the machine
If you have a new PC, you might want to consider installing the Fermilab-enhanced Linux. The image includes UPS/UPD . Go to http://fermilinux.fnal.gov/ .
Otherwise, consider using tar as described earlier or go to Chapter 14: Installing UPS and UPD from Bootstrap and follow the directions there. We recommend choosing one of the predefined UPS/UPD configurations.
1.5 Something is Wrong¶
Before you can set up a product, you need to be on the right machine and have ups set up correctly. Otherwise, the following may happen:
%setup myproduct You are attempting to run "setup" which requires administrative privileges, but more information is needed in order to do so. Authenticating as "root"
If this happens, contact someone in your experiment and verify what General Physics Compute Facility (GPCF) machine you should be using, and where the setup script is.
p<>. If you're on the right machine, and have ups set up correctly, but type the name wrong, you will get the following error message:
%setup myproduct ERROR: Product 'myproduct' (with qualifiers ''), has no current chain (or may not exist)
For much more information on UPS and UPD, keep reading this guide.
1.6 Checking Layout Before ups V5_1¶
For ups versions before v5_1, the following can be used to determine if the directory structure of the source and destination product areas are the same.
% ups list -Kprod_dir_prefix:db
If the two answers are the same, then this tar method can be used.
p<>. An answer like the following means tar can be used:
$ ups list -Kprod_dir_prefix:db -z /nusoft/app/externals "/nusoft/app/externals" "/nusoft/app/externals"
But an answer with different values means tar cannot be used.
Cannot use tar: $ ups list -Kprod_dir_prefix:db -z /grid/fermiapp/products/common/db "/grid/fermiapp/products/common/db/../prd" "/grid/fermiapp/products/common/db"
| |
|
|
This page last revised in May 2014
| |
|
|
| |
Complete Guide and Reference Manual for UPS and UPD | |||
Chapter Contents¶
Chapter 1: Quick! I Just Want to ...
1.1 Why UPS/UPD is Needed
1.2 Install a Product using tar
1.3 Install a Product from KITS; No UPS/UPD on Target Machine
1.4 Install UPS/UPD in a Preset Configuration
1.5 Something is Wrong
1.6 Checking Layout Before ups V5_1
Chapter 1: Quick! I Just Want to ...¶
This chapter is targeted to people who don't know anything about Unix Product Support and Unix Product Distribution UPS/UPD but want to get started quickly without having to read very much.
If you already know what to install, and where, you can skip to section 1.2 Install a Product using tar
1.1 Why UPS/UPD is Needed¶
The software for experiments rely on external tools. These packages and tools are referred to as external products, or simply products. The following is just a partial list, and may not apply to all experiments. It is given to illustrate the types of packages and tools that are external products.
- art -- the event processing framework
- CLHEP -- Class Library for High Energy Physics
- Geant4 -- toolkit for the simulation of the passage of particles through matter.
- git -- a source code management system
- ROOT -- an object oriented framework for large scale data analysis
- UPD -- Unix Product Distribution - used to distribute software products
- UPS -- Unix Product Support -- used to access software products
For Fermilab-based experiments, access to external products is provided by a Fermilab-developed product-management package called Unix Product Support. UPS supports multiple versions of a product and multiple builds per version. For example, a product multiple times for use with different versions of an operating system. A product could be built with a production version of the C++ complier, and with a version under test. It could be built with full compiler optimization or with the maximum debugging features enabled. UPS provides a way to select a particular build. The full identifier of a UPS product includes:
- product name
- version
- flavor
- full set of qualifiers
An important UPS feature, demanded by most experiments as their code evolved, is support for multiple versions of a product and multiple builds (e.g., for different platforms) per version. Another notable feature is the capacity to handle multiple databases of products. On Fermilab computers, login scripts set up the UPS system, providing access to a database of products commonly used at Fermilab.
The act of setting up UPS defines a number of environment variables, one of which is PRODUCTS, a colon-delimited list of directory names. Each directory in PRODUCTS is the name of a UPS database, meaning each directory functions as a repository of information about one or more products. When UPS looks for a product, it checks each directory in PRODUCTS in the order listed, and takes the first match.
To see an example of a UPS repository, the command printenv PRODUCTS (on fermicloud050), gives the directory /fnal/ups/db. The command ls will list the products found there.
$ printenv PRODUCTS /fnal/ups/db $ ls /fnal/ups/db catman fnal_git_notifier lscvs python subversion ups cvs git lxr redmine upd wu_ftpd cvsh ifdhc man ruby upd_libs xxx
For a second example, on novagpvm02 in early 2014, the printenv command gives two directories, with the following products.
printenv PRODUCTS /nusoft/app/externals:/grid/fermiapp/products/common/db % ls /nusoft/app/externals AAAREADME fhiclcpp lhapdf pqxx allinea_tools fluka libsigcpp psycopg2 (and many more) % ls /grid/fermiapp/products/common/db afs ifdhc mako project twisted cherrypy jobsub minos pycurl upd (and many more)
UPS can make multiple variants of a product available to users. This includes different versions, but beyond that, a given version of a product may be built more than one way, e.g., for use by different operating systems (what UPS distinguishes as flavors). Many variants can exist. UPS provides a way to select a particular build via qualifiers. The full identifier of a UPS product includes its product name, its version, its flavor and its full set of qualifiers.
For some UPS products, but not all, the site administrator may define a particular fully-qualified version of the product as the default version. In the language of UPS this notion of default is called the current version. If a current version has been defined for a product, users can set up that product with the command:
$ setup <product>
The UPS system will automatically insert the version and qualifiers of the version that has been declared current. Having a current version is a handy feature since as improvements are added, users automatically get them.
p<>. However the notion of a current version is very dangerous if users want to ensure that software built at one site will build in exactly the same way on all other sites. For this reason, some projects require fully specifying the version number and qualifiers of all products that are required; and in turn, the products make fully qualified requests for the products on which they depend.
Many software products have version numbers that contain dot characters. UPS requires that version numbers not contain any dot characters; by convention, version dots are replaced with underscores. Therefore v0.00.09 becomes v0_00_09. Also by convention, the environment variables are all upper case, regardless of the case used in the product names.
1.2 Install a Product using tar¶
p<>.
The UNIX command tar can be used to combine files into a single file, for easy distribution.
Using tar requires UPS version 5 or later. To use the tar command, the directory structure of the source and destination product areas must be the same. To check this in ups v5_1 or later, run:
% ups layout
If it returns 'unified', then tar can be used. In the following output:
"/scratch/mengel/p": "unified" "/fnal/ups/db": "traditional" "/afs/fnal.gov/ups/db": "traditional"
tar can be used for /scratch/mengel/p.
p<>. For earlier versions of ups , see Checking Layout Before ups V5_1
Once the layout has been verified, the user changes directory to the products area and runs the tar command with the c (create) option.
% cd /products/area % tar czvf /tmp/transfer_me.tgz product1 product2/vx_y_z*
This command packs up all versions of product1 and version vx_y_z of product2 into /tmp/transfer_me.tgz. The tarfile can be transferred however the user wishes to get it to the machine to install it on, such as Google Docs, put it on a webserver, etc. The user changes directory to the products area and runs the tar command with the x (extract) option to get the files.
% cd /other/products/area % tar xzvf /tmp/transfer_me.tgz
Users can also extract only particular packages from the tarfile, by listing exactly what is wanted.
% tar xzvf /tmp/transfer_me.tgz product1
For details on using tar to install a product, please see Chapter 7 .
1.3 Install a Product from KITS; UPS/UPD Exists on Target Machine¶
p<>.
The KITS software repository is available to all computers in the fnal.gov domain and registered users. If you think you don't need to know anything about KITS, be aware that low-level data-handling software is often distributed through UPD, and UPD uses KITS. You may well have scripts that call UPD, which uses KITS.
This section assumes:
- UPS/UPD is installed and configured properly on the target machine, and the defaults are set appropriately
- the product from KITS is to be installed into the standard UPS product area on the machine
- a product tar or zip file made for the OS exists in KITS, and is set as "current" so the version or flavor isn't needed, just the product name
- the product is configured in KITS such that it can be installed in a straightforward manner
On your machine, first setup UPD and verify that the product you want is available from KITS. To do so, issue the following commands:
% setup upd % upd list <product>
You should see output in the format:
DATABASE=/ftp/upsdb
Product=lscvs Version=v1_1 Flavor=NULL
Qualifiers="" Chain=current
If there is no output, or only the DATABASE variable is given, either there's no instance of the product for the OS, or there is one but it's not set to "current". See 24.9 upd list for other options to include.
Assuming the product is there, the next step is to install it. To see where UPS/UPD will install it, first run the command:
% upd install -sv <product>
which will not install it, but will output information about what it would do to install it. Look for the directory it would create for installing the product. Also look for the ups(local) declare... command; the argument to the -z options tells you the database UPS would use to declare the product.
A few notes:
- If UPS/UPD wants to install into AFS space, you can't do a "quick and dirty" install. Check with someone more UPS -savvy and/or AFS-savvy to see if this is the right destination for the product; if so, refer to Chapter 6: Installing Products Using UPD and section 9.3 Installing Products into AFS Space .
- If you want to check if there are databases choices, echo $PRODUCTS to see if there's multiple database paths listed. If so, you can redefine $PRODUCTS such that the one you want is first, then rerun the upd install -sv <product> command to see if the result would change.
- If you want the product in your own area, declared in your own database, again you'll have to know more. See Chapter 6: Installing Products Using UPD and section 12.8.2 Adding a New Database and/or Products Area .
Assuming the upd install -sv command output is what you want, install the product. To do so, enter the command:
% upd install -G "-c" <product>
This installs the product so only setup <product> , is needed before the product is invoked. If you're not sure what command name to use to invoke the product, look for the executable name under $<PRODUCT>_DIR/bin/ .
More information is available in Chapter 6: Installing Products Using UPD . There is a checklist in section 6.4 Checklist for Installing a Product using UPD and examples in section 6.5 Examples .
1.4 Install UPS/UPD in a Preset Configuration¶
This section assumes:
- UPS/UPD is not installed on the target machine, but you want to install it using a simple, preset configuration
- you want to install products from KITS on the machine
If you have a new PC, you might want to consider installing the Fermilab-enhanced Linux. The image includes UPS/UPD . Go to http://fermilinux.fnal.gov/ .
Otherwise, consider using tar as described earlier or go to Chapter 14: Installing UPS and UPD from Bootstrap and follow the directions there. We recommend choosing one of the predefined UPS/UPD configurations.
1.5 Something is Wrong¶
Before you can set up a product, you need to be on the right machine and have ups set up correctly. Otherwise, the following may happen:
%setup myproduct You are attempting to run "setup" which requires administrative privileges, but more information is needed in order to do so. Authenticating as "root"
If this happens, contact someone in your experiment and verify what General Physics Compute Facility (GPCF) machine you should be using, and where the setup script is.
p<>. If you're on the right machine, and have ups set up correctly, but type the name wrong, you will get the following error message:
%setup myproduct ERROR: Product 'myproduct' (with qualifiers ''), has no current chain (or may not exist)
For much more information on UPS and UPD, keep reading this guide.
1.6 Checking Layout Before ups V5_1¶
For ups versions before v5_1, the following can be used to determine if the directory structure of the source and destination product areas are the same.
% ups list -Kprod_dir_prefix:db
If the two answers are the same, then this tar method can be used.
p<>. An answer like the following means tar can be used:
$ ups list -Kprod_dir_prefix:db -z /nusoft/app/externals "/nusoft/app/externals" "/nusoft/app/externals"
But an answer with different values means tar cannot be used.
Cannot use tar: $ ups list -Kprod_dir_prefix:db -z /grid/fermiapp/products/common/db "/grid/fermiapp/products/common/db/../prd" "/grid/fermiapp/products/common/db"
| |
|
|
This page last revised in May 2014