Project

General

Profile

TOC PREV NEXT
Fermilab CD logo Complete Guide and Reference Manual for UPS and UPD

Chapter Contents

Chapter 1: Quick! I Just Want to ...
   1.1 Why UPS/UPD is Needed
   1.2 Install a Product using tar
   1.3 Install a Product from KITS; No UPS/UPD on Target Machine
   1.4 Install UPS/UPD in a Preset Configuration
   1.5 Something is Wrong
   1.6 Checking Layout Before ups V5_1

  Chapter 1: Quick! I Just Want to ...

  This chapter is targeted to people who don't know anything about Unix Product Support and Unix Product Distribution UPS/UPD but want to get started quickly without having to read very much.

If you already know what to install, and where, you can skip to section 1.2 Install a Product using tar

  1.1 Why UPS/UPD is Needed

 
The software for experiments rely on external tools. These packages and tools are referred to as external products, or simply products. The following is just a partial list, and may not apply to all experiments. It is given to illustrate the types of packages and tools that are external products.

  • art -- the event processing framework
  • CLHEP -- Class Library for High Energy Physics
  • Geant4 -- toolkit for the simulation of the passage of particles through matter.
  • git -- a source code management system
  • ROOT -- an object oriented framework for large scale data analysis
  • UPD -- Unix Product Distribution - used to distribute software products
  • UPS -- Unix Product Support -- used to access software products

For Fermilab-based experiments, access to external products is provided by a Fermilab-developed product-management package called Unix Product Support. UPS supports multiple versions of a product and multiple builds per version. For example, a product multiple times for use with different versions of an operating system. A product could be built with a production version of the C++ complier, and with a version under test. It could be built with full compiler optimization or with the maximum debugging features enabled. UPS provides a way to select a particular build. The full identifier of a UPS product includes:

  • product name
  • version
  • flavor
  • full set of qualifiers

An important UPS feature, demanded by most experiments as their code evolved, is support for multiple versions of a product and multiple builds (e.g., for different platforms) per version. Another notable feature is the capacity to handle multiple databases of products. On Fermilab computers, login scripts set up the UPS system, providing access to a database of products commonly used at Fermilab.

The act of setting up UPS defines a number of environment variables, one of which is PRODUCTS, a colon-delimited list of directory names. Each directory in PRODUCTS is the name of a UPS database, meaning each directory functions as a repository of information about one or more products. When UPS looks for a product, it checks each directory in PRODUCTS in the order listed, and takes the first match.

To see an example of a UPS repository, the command printenv PRODUCTS (on fermicloud050), gives the directory /fnal/ups/db. The command ls will list the products found there.

$ printenv PRODUCTS
/fnal/ups/db

$ ls /fnal/ups/db
catman  fnal_git_notifier  lscvs  python   subversion  ups
cvs     git                lxr    redmine  upd         wu_ftpd
cvsh    ifdhc              man    ruby     upd_libs    xxx

For a second example, on novagpvm02 in early 2014, the printenv command gives two directories, with the following products.

printenv PRODUCTS
/nusoft/app/externals:/grid/fermiapp/products/common/db

% ls /nusoft/app/externals
AAAREADME      fhiclcpp          lhapdf           pqxx
allinea_tools  fluka             libsigcpp        psycopg2
(and many more)

% ls /grid/fermiapp/products/common/db
afs           ifdhc             mako    project   twisted
cherrypy  jobsub        minos   pycurl    upd
(and many more)

UPS can make multiple variants of a product available to users. This includes different versions, but beyond that, a given version of a product may be built more than one way, e.g., for use by different operating systems (what UPS distinguishes as flavors). Many variants can exist. UPS provides a way to select a particular build via qualifiers. The full identifier of a UPS product includes its product name, its version, its flavor and its full set of qualifiers.

For some UPS products, but not all, the site administrator may define a particular fully-qualified version of the product as the default version. In the language of UPS this notion of default is called the current version. If a current version has been defined for a product, users can set up that product with the command:

$ setup <product>

The UPS system will automatically insert the version and qualifiers of the version that has been declared current. Having a current version is a handy feature since as improvements are added, users automatically get them.
p<>. However the notion of a current version is very dangerous if users want to ensure that software built at one site will build in exactly the same way on all other sites. For this reason, some projects require fully specifying the version number and qualifiers of all products that are required; and in turn, the products make fully qualified requests for the products on which they depend.

Many software products have version numbers that contain dot characters. UPS requires that version numbers not contain any dot characters; by convention, version dots are replaced with underscores. Therefore v0.00.09 becomes v0_00_09. Also by convention, the environment variables are all upper case, regardless of the case used in the product names.

  1.2 Install a Product using tar

p<>.

The UNIX command tar can be used to combine files into a single file, for easy distribution.

Using tar requires UPS version 5 or later. To use the tar command, the directory structure of the source and destination product areas must be the same. To check this in ups v5_1 or later, run:

% ups layout

If it returns 'unified', then tar can be used. In the following output:

"/scratch/mengel/p": "unified" 
"/fnal/ups/db": "traditional" 
"/afs/fnal.gov/ups/db": "traditional" 

tar can be used for /scratch/mengel/p.
p<>. For earlier versions of ups , see Checking Layout Before ups V5_1

Once the layout has been verified, the user changes directory to the products area and runs the tar command with the c (create) option.

% cd /products/area
% tar czvf /tmp/transfer_me.tgz  product1 product2/vx_y_z*

This command packs up all versions of product1 and version vx_y_z of product2 into /tmp/transfer_me.tgz. The tarfile can be transferred however the user wishes to get it to the machine to install it on, such as Google Docs, put it on a webserver, etc. The user changes directory to the products area and runs the tar command with the x (extract) option to get the files.

% cd /other/products/area
% tar xzvf /tmp/transfer_me.tgz

Users can also extract only particular packages from the tarfile, by listing exactly what is wanted.

% tar xzvf /tmp/transfer_me.tgz product1

For details on using tar to install a product, please see Chapter 7 .

  1.3 Install a Product from KITS; UPS/UPD Exists on Target Machine

p<>.

The KITS software repository is available to all computers in the fnal.gov domain and registered users. If you think you don't need to know anything about KITS, be aware that low-level data-handling software is often distributed through UPD, and UPD uses KITS. You may well have scripts that call UPD, which uses KITS.
This section assumes:

  • UPS/UPD is installed and configured properly on the target machine, and the defaults are set appropriately
  • the product from KITS is to be installed into the standard UPS product area on the machine
  • a product tar or zip file made for the OS exists in KITS, and is set as "current" so the version or flavor isn't needed, just the product name
  • the product is configured in KITS such that it can be installed in a straightforward manner

  On your machine, first setup UPD and verify that the product you want is available from KITS. To do so, issue the following commands:

% setup upd
% upd list <product>

  You should see output in the format:

DATABASE=/ftp/upsdb
        Product=lscvs   Version=v1_1    Flavor=NULL
                Qualifiers=""   Chain=current

  If there is no output, or only the DATABASE variable is given, either there's no instance of the product for the OS, or there is one but it's not set to "current". See 24.9 upd list for other options to include.

  Assuming the product is there, the next step is to install it. To see where UPS/UPD will install it, first run the command:

% upd install -sv <product>

  which will not install it, but will output information about what it would do to install it. Look for the directory it would create for installing the product. Also look for the ups(local) declare... command; the argument to the -z options tells you the database UPS would use to declare the product.

  A few notes:

  Assuming the upd install -sv command output is what you want, install the product. To do so, enter the command:

% upd install -G "-c" <product>

  This installs the product so only setup <product> , is needed before the product is invoked. If you're not sure what command name to use to invoke the product, look for the executable name under $<PRODUCT>_DIR/bin/ .

  More information is available in Chapter 6: Installing Products Using UPD . There is a checklist in section 6.4 Checklist for Installing a Product using UPD and examples in section 6.5 Examples .

  1.4 Install UPS/UPD in a Preset Configuration

  This section assumes:

  • UPS/UPD is not installed on the target machine, but you want to install it using a simple, preset configuration
  • you want to install products from KITS on the machine

  If you have a new PC, you might want to consider installing the Fermilab-enhanced Linux. The image includes UPS/UPD . Go to http://fermilinux.fnal.gov/ .

  Otherwise, consider using tar as described earlier or go to Chapter 14: Installing UPS and UPD from Bootstrap and follow the directions there. We recommend choosing one of the predefined UPS/UPD configurations.

  1.5 Something is Wrong

Before you can set up a product, you need to be on the right machine and have ups set up correctly. Otherwise, the following may happen:

%setup myproduct

You are attempting to run "setup" which requires administrative
privileges, but more information is needed in order to do so.
Authenticating as "root" 

If this happens, contact someone in your experiment and verify what General Physics Compute Facility (GPCF) machine you should be using, and where the setup script is.
p<>. If you're on the right machine, and have ups set up correctly, but type the name wrong, you will get the following error message:

%setup myproduct

ERROR: Product 'myproduct' (with qualifiers ''), has no current chain 
(or may not exist)

For much more information on UPS and UPD, keep reading this guide.

  1.6 Checking Layout Before ups V5_1

 

For ups versions before v5_1, the following can be used to determine if the directory structure of the source and destination product areas are the same.

% ups list -Kprod_dir_prefix:db

If the two answers are the same, then this tar method can be used.
p<>. An answer like the following means tar can be used:

$  ups list -Kprod_dir_prefix:db -z /nusoft/app/externals
"/nusoft/app/externals" "/nusoft/app/externals" 

But an answer with different values means tar cannot be used.

Cannot use tar:
$ ups list -Kprod_dir_prefix:db -z /grid/fermiapp/products/common/db
"/grid/fermiapp/products/common/db/../prd" "/grid/fermiapp/products/common/db" 
TOC PREV NEXT

This page last revised in May 2014

TOC PREV NEXT
Fermilab CD logo Complete Guide and Reference Manual for UPS and UPD

Chapter Contents

Chapter 1: Quick! I Just Want to ...
   1.1 Why UPS/UPD is Needed
   1.2 Install a Product using tar
   1.3 Install a Product from KITS; No UPS/UPD on Target Machine
   1.4 Install UPS/UPD in a Preset Configuration
   1.5 Something is Wrong
   1.6 Checking Layout Before ups V5_1

  Chapter 1: Quick! I Just Want to ...

  This chapter is targeted to people who don't know anything about Unix Product Support and Unix Product Distribution UPS/UPD but want to get started quickly without having to read very much.

If you already know what to install, and where, you can skip to section 1.2 Install a Product using tar

  1.1 Why UPS/UPD is Needed

 
The software for experiments rely on external tools. These packages and tools are referred to as external products, or simply products. The following is just a partial list, and may not apply to all experiments. It is given to illustrate the types of packages and tools that are external products.

  • art -- the event processing framework
  • CLHEP -- Class Library for High Energy Physics
  • Geant4 -- toolkit for the simulation of the passage of particles through matter.
  • git -- a source code management system
  • ROOT -- an object oriented framework for large scale data analysis
  • UPD -- Unix Product Distribution - used to distribute software products
  • UPS -- Unix Product Support -- used to access software products

For Fermilab-based experiments, access to external products is provided by a Fermilab-developed product-management package called Unix Product Support. UPS supports multiple versions of a product and multiple builds per version. For example, a product multiple times for use with different versions of an operating system. A product could be built with a production version of the C++ complier, and with a version under test. It could be built with full compiler optimization or with the maximum debugging features enabled. UPS provides a way to select a particular build. The full identifier of a UPS product includes:

  • product name
  • version
  • flavor
  • full set of qualifiers

An important UPS feature, demanded by most experiments as their code evolved, is support for multiple versions of a product and multiple builds (e.g., for different platforms) per version. Another notable feature is the capacity to handle multiple databases of products. On Fermilab computers, login scripts set up the UPS system, providing access to a database of products commonly used at Fermilab.

The act of setting up UPS defines a number of environment variables, one of which is PRODUCTS, a colon-delimited list of directory names. Each directory in PRODUCTS is the name of a UPS database, meaning each directory functions as a repository of information about one or more products. When UPS looks for a product, it checks each directory in PRODUCTS in the order listed, and takes the first match.

To see an example of a UPS repository, the command printenv PRODUCTS (on fermicloud050), gives the directory /fnal/ups/db. The command ls will list the products found there.

$ printenv PRODUCTS
/fnal/ups/db

$ ls /fnal/ups/db
catman  fnal_git_notifier  lscvs  python   subversion  ups
cvs     git                lxr    redmine  upd         wu_ftpd
cvsh    ifdhc              man    ruby     upd_libs    xxx

For a second example, on novagpvm02 in early 2014, the printenv command gives two directories, with the following products.

printenv PRODUCTS
/nusoft/app/externals:/grid/fermiapp/products/common/db

% ls /nusoft/app/externals
AAAREADME      fhiclcpp          lhapdf           pqxx
allinea_tools  fluka             libsigcpp        psycopg2
(and many more)

% ls /grid/fermiapp/products/common/db
afs           ifdhc             mako    project   twisted
cherrypy  jobsub        minos   pycurl    upd
(and many more)

UPS can make multiple variants of a product available to users. This includes different versions, but beyond that, a given version of a product may be built more than one way, e.g., for use by different operating systems (what UPS distinguishes as flavors). Many variants can exist. UPS provides a way to select a particular build via qualifiers. The full identifier of a UPS product includes its product name, its version, its flavor and its full set of qualifiers.

For some UPS products, but not all, the site administrator may define a particular fully-qualified version of the product as the default version. In the language of UPS this notion of default is called the current version. If a current version has been defined for a product, users can set up that product with the command:

$ setup <product>

The UPS system will automatically insert the version and qualifiers of the version that has been declared current. Having a current version is a handy feature since as improvements are added, users automatically get them.
p<>. However the notion of a current version is very dangerous if users want to ensure that software built at one site will build in exactly the same way on all other sites. For this reason, some projects require fully specifying the version number and qualifiers of all products that are required; and in turn, the products make fully qualified requests for the products on which they depend.

Many software products have version numbers that contain dot characters. UPS requires that version numbers not contain any dot characters; by convention, version dots are replaced with underscores. Therefore v0.00.09 becomes v0_00_09. Also by convention, the environment variables are all upper case, regardless of the case used in the product names.

  1.2 Install a Product using tar

p<>.

The UNIX command tar can be used to combine files into a single file, for easy distribution.

Using tar requires UPS version 5 or later. To use the tar command, the directory structure of the source and destination product areas must be the same. To check this in ups v5_1 or later, run:

% ups layout

If it returns 'unified', then tar can be used. In the following output:

"/scratch/mengel/p": "unified" 
"/fnal/ups/db": "traditional" 
"/afs/fnal.gov/ups/db": "traditional" 

tar can be used for /scratch/mengel/p.
p<>. For earlier versions of ups , see Checking Layout Before ups V5_1

Once the layout has been verified, the user changes directory to the products area and runs the tar command with the c (create) option.

% cd /products/area
% tar czvf /tmp/transfer_me.tgz  product1 product2/vx_y_z*

This command packs up all versions of product1 and version vx_y_z of product2 into /tmp/transfer_me.tgz. The tarfile can be transferred however the user wishes to get it to the machine to install it on, such as Google Docs, put it on a webserver, etc. The user changes directory to the products area and runs the tar command with the x (extract) option to get the files.

% cd /other/products/area
% tar xzvf /tmp/transfer_me.tgz

Users can also extract only particular packages from the tarfile, by listing exactly what is wanted.

% tar xzvf /tmp/transfer_me.tgz product1

For details on using tar to install a product, please see Chapter 7 .

  1.3 Install a Product from KITS; UPS/UPD Exists on Target Machine

p<>.

The KITS software repository is available to all computers in the fnal.gov domain and registered users. If you think you don't need to know anything about KITS, be aware that low-level data-handling software is often distributed through UPD, and UPD uses KITS. You may well have scripts that call UPD, which uses KITS.
This section assumes:

  • UPS/UPD is installed and configured properly on the target machine, and the defaults are set appropriately
  • the product from KITS is to be installed into the standard UPS product area on the machine
  • a product tar or zip file made for the OS exists in KITS, and is set as "current" so the version or flavor isn't needed, just the product name
  • the product is configured in KITS such that it can be installed in a straightforward manner

  On your machine, first setup UPD and verify that the product you want is available from KITS. To do so, issue the following commands:

% setup upd
% upd list <product>

  You should see output in the format:

DATABASE=/ftp/upsdb
        Product=lscvs   Version=v1_1    Flavor=NULL
                Qualifiers=""   Chain=current

  If there is no output, or only the DATABASE variable is given, either there's no instance of the product for the OS, or there is one but it's not set to "current". See 24.9 upd list for other options to include.

  Assuming the product is there, the next step is to install it. To see where UPS/UPD will install it, first run the command:

% upd install -sv <product>

  which will not install it, but will output information about what it would do to install it. Look for the directory it would create for installing the product. Also look for the ups(local) declare... command; the argument to the -z options tells you the database UPS would use to declare the product.

  A few notes:

  Assuming the upd install -sv command output is what you want, install the product. To do so, enter the command:

% upd install -G "-c" <product>

  This installs the product so only setup <product> , is needed before the product is invoked. If you're not sure what command name to use to invoke the product, look for the executable name under $<PRODUCT>_DIR/bin/ .

  More information is available in Chapter 6: Installing Products Using UPD . There is a checklist in section 6.4 Checklist for Installing a Product using UPD and examples in section 6.5 Examples .

  1.4 Install UPS/UPD in a Preset Configuration

  This section assumes:

  • UPS/UPD is not installed on the target machine, but you want to install it using a simple, preset configuration
  • you want to install products from KITS on the machine

  If you have a new PC, you might want to consider installing the Fermilab-enhanced Linux. The image includes UPS/UPD . Go to http://fermilinux.fnal.gov/ .

  Otherwise, consider using tar as described earlier or go to Chapter 14: Installing UPS and UPD from Bootstrap and follow the directions there. We recommend choosing one of the predefined UPS/UPD configurations.

  1.5 Something is Wrong

Before you can set up a product, you need to be on the right machine and have ups set up correctly. Otherwise, the following may happen:

%setup myproduct

You are attempting to run "setup" which requires administrative
privileges, but more information is needed in order to do so.
Authenticating as "root" 

If this happens, contact someone in your experiment and verify what General Physics Compute Facility (GPCF) machine you should be using, and where the setup script is.
p<>. If you're on the right machine, and have ups set up correctly, but type the name wrong, you will get the following error message:

%setup myproduct

ERROR: Product 'myproduct' (with qualifiers ''), has no current chain 
(or may not exist)

For much more information on UPS and UPD, keep reading this guide.

  1.6 Checking Layout Before ups V5_1

 

For ups versions before v5_1, the following can be used to determine if the directory structure of the source and destination product areas are the same.

% ups list -Kprod_dir_prefix:db

If the two answers are the same, then this tar method can be used.
p<>. An answer like the following means tar can be used:

$  ups list -Kprod_dir_prefix:db -z /nusoft/app/externals
"/nusoft/app/externals" "/nusoft/app/externals" 

But an answer with different values means tar cannot be used.

Cannot use tar:
$ ups list -Kprod_dir_prefix:db -z /grid/fermiapp/products/common/db
"/grid/fermiapp/products/common/db/../prd" "/grid/fermiapp/products/common/db" 
TOC PREV NEXT

This page last revised in May 2014