Project

General

Profile

Getting Started Using UPS

A product-management package called Unix Product Support (UPS), developed at Fermilab, provides access to software products commonly used by Fermilab experiments. UPS can handle multiple databases of products and multiple versions of products. Login Scripts typically set up the UPS system. Users are not always aware they are using UPS since it may be in setup scripts called by experiment code.

There are often multiple versions of a software product, and Software Configuration Management is used to track and control changes in software as part of Configuration Management

Environment Variables Defined by UPS

Products need to be setup to be used -- again, often done within scripts. This process, discussed below, defines a number of environment variables, one of which is PRODUCTS. PRODUCTS is a colon-delimited list of directory names. Each directory, also known as a UPS database, functions as a repository of information about one or more products. When UPS looks for a product, it checks each directory in PRODUCTS, in the order listed, and takes the first match.

On a Fermilab machine, the value of PRODUCTS (before any login scripts are run) can be seen as follows:

% printenv PRODUCTS
/grid/fermiapp/products/common/db

This generic Fermilab UPS database contains a handful of software products commonly used by Fermilab experiments. This database does not contain any of the experiment-specific software nor does it contain products that are often custom-built for each experiment and as such are distributed via the experiment-specific (i.e., separate) UPS databases.

To get experiment-specific UPS databases, you need to execute a setup shell script specific to the experiment. This example was done on novagpvm02 in June of 2014.

 % . /fnal/ups/etc/setups.sh
 % printenv PRODUCTS
/fnal/ups/db:/afs/fnal/ups/db

PRODUCTS now contains multiple directories, thus making more products available. These can be seen by listing the contents of the directory.

Many software products have version numbers that contain dot characters. UPS requires that version numbers not contain any dot characters; therefore UPS replaces v0.00.09 by v0_00_09, for example.

Many other variables are set, these can be seen via the printenv command with no parameters.

% printenv

In particular, for a product <product_name>:

  1. <PRODUCT_NAME>_DIR defines the root directory of the chosen UPS product.
  2. For most products that are used in analysis, <PRODUCT_NAME>_INC defines the path to the root directory of the C++ header files that are provided by the product (so called because the header files are INCluded), and
  3. <PRODUCT_NAME>_LIB defines the directory that contains all of the shareable object LIBraries (ending in .so) that are provided by the product.

To see all the environment variables associated with the product, type:

% printenv | grep <PRODUCT_NAME>

By convention, the environment variables are always in upper case, regardless of the case used in the product names; e.g., a product called myproduct would have the associated environment variable MYPRODUCT_DIR.

Products Available via UPS

Each directory name in the database listing corresponds to the name of a UPS product. You can see the products via the ls command.

 % . /fnal/ups/etc/setups.sh
 % printenv PRODUCTS
/fnal/ups/db:/afs/fnal/ups/db
 % ls /fnal/upd/db
 % ls /afs/fnalups/db

For example:

% ls /fnal/ups/db

bash        cmdscriptsbase  coreFUE  perl     stock       template_home
cmdbin        cmdscriptsctrl  flpr     setpath  subm       upd
cmdbincmd   cmdscriptsguid  fullFUE  setterm  systools       ups
cmdbingen   cmdscriptshell  futil    setxwin  systoolsutl  xxx
cmdbinutl   cmdscriptslgrp  Info     shells   tcsh
cmdscripts  cmdscriptsuser  login    shrc     telephone

% ls /afs/fnalups/db
(has even more products)

You can also see products via the UPS command,

% ups list -K+

where -K+ provides the list in a condensed format. Details are available in the UPS Documentation.

The precise contents of an experiment’s product directory depends on the experiment. Among other things, each database contains a subdirectory named ups. This is for the UPS system itself. In this sense, UPS is just another external product.

How UPS handles variants of a given product

An important feature of UPS is its capacity to make multiple variants of a product available to users. This includes different flavors (Operating Systems) and qualifiers (e.g., different compilers and/or different compiler options). A given version of a product may be built in more than one way.

For example, a version of a product might be built once for use with SLF5 and again for use with SLF6. A product may be built with different versions of the C++ compiler, e.g., with the production version and with a version under test. A product may be built with full compiler optimization or with the maximum debugging features enabled. Many variants can exist.

UPS provides a way to select a particular build via qualifiers.
The full identifier of a UPS product includes
  1. product name,
  2. version,
  3. flavor,
  4. full set of qualifiers

The product name is either known or can be found as shown above. The version can be determined via:

% ups list -K product:version 

This (on novagpvm02) produces:

"allinea_tools" "v4_1_1" 
"cetpkgsupport" "v1_04_02" 
"cmake" "v2_8_8" 
"git" "v1_8_0_1" 
"sam_web_client" "v1_3" 
"totalview" "v8_11_0" 
"upd" "v4_7_5" 
(and many more)

The flavor is the Operating System specification of the machine. When entered with no options, the command returns the full OS specification of the machine, for example:

% ups flavor
Linux64bit+2.6-2.12

When entered with the -l (long) option, it returns a flavor table, which takes the basic flavor and lists it at every level of specificity to find or declare a product instance. For example:

% ups flavor -l 

Linux64bit+2.6-2.12 
Linux64bit+2.6-2.11 
Linux64bit+2.6-2.10 
(many more) 
Linux+2.6 
(many more) 
Linux 
NULL 
ANY

More information is available in the UPS/UPD Documentation

Strict Dependencies, -B flag

If the dependencies for a package involve different versions of the same package, UPS chooses the first one it sees, by default. However, with the -B flag, which enforces strict dependencies, it is an error to have conflicts in dependencies. This is to ensure that the exact version and qualifiers are requested to avoid having a default qualifier change in a poorly documented fashion, making it difficult to reproduce results.

The -B flag affects dependencies so that having different branches of the dependency tree end in different instances of the same product is an error. Attempting to set up something that differs from what is already set up is an error. Users must first unsetup to switch versions. This flag can be set in the environment variable UPS_OVERRIDE, which
means it affects all ups commands.

How to use UPS to Initialize a Product

Any given UPS database contains many products. To select a product and make it available for use, use the setup command.

For some products, the correct flavor can be automatically detected by setup and need not be specified. However, if needed, flavor, in addition to various qualifiers and options can be specified. The product version, if specified, must directly follow the product name in the command line:

$ setup <options> <product-name> <product-version> -f <flavor> -q <qualifiers>

Putting in real-looking values, it would look something like:

$ setup -R myproduct vNumber  -f OperatingSystem -q BUILD_A

The setup command:
  1. defines environment variables
  2. defines bash functions
  3. adds elements to your PATH
  4. sets up additional products on which it depends, if any

Setting up dependent products works recursively. In this way, a single setup command may trigger the setup of multiple products.

When you follow a given site-specific setup procedure, the PRODUCTS environment variable will be extended to include your experiment-specific UPS repository.

setup is a bash function (defined by the UPS product when it was initialized) that shadows a Unix system-configuration command also named setup, usually found in /usr/bin/setup or /usr/sbin/setup. Running the right ‘setup’ should work automatically as long as UPS is properly initialized. If it’s not, setup returns the error message:

You are attempting to run ‘‘setup'' which requires administrative  
privileges, but more information is needed in order to do so.

If this happens, the simplest solution is to log out and log in again. If the error continues to occur, consult your experiment to ensure you are running the proper login scripts.

For more information, consult the full UPS documentation

Current Versions of Products

For some UPS products, the site administrator may define a particular fully-qualified version of the product as the default version. This is called the current version.

If a current version has been defined for a product, you can set up that product with the command:

$ setup <product-name>

When run, the UPS system automatically inserts the version and qualifiers of the version that has been declared current. Having a current version is a handy convenience for products that add features in the environment; as improvements are added, you automatically get them.

However the notion of a current version can be dangerous if you want to ensure that software built at one site will build in exactly the same way on all other sites. Some experiments specify the version number and qualifiers of all products that are experiment-data-affecting that it requires.

For example, to setup an instance of art, the command is:

% setup -B art v1_08_09 -q+e4:+mu2e:+prof

The version is specified as v1_08_09. Qualifiers are used to keep the bookkeeping straight for multiple builds of one version of a product. e4 is shorthand defined by the art team for a compiler version and other specifics. When another product is built with the "e4" qualifier, it can be linked against a version of art that was also built with the "e4" qualifier. The qualifier "prof" requests a version that was built with maximum optimization and with some, but not all, debugger symbols retained. The symbols retained are enough to profile the code and to get a traceback from a core dump. (But not enough to do full line by line debugging.) The name "prof" is motivated by the notion that the build retains enough debugger symbols for profiling.

For more details, please see: The setup Command with -q Option in the UPS Documentation

UPS Documentation

The complete manual is available here