Project

General

Profile

TOC PREV NEXT
Fermilab CD logo Complete Guide and Reference Manual for UPS and UPD

Chapter Contents

Chapter 4: Product Installation Basics
   4.1 Installation Methods for UPS Products
     4.1.1 tar
     4.1.2 UPD
     4.1.3 Installing UPD Products by Hand
   4.2 Modifying Your UPD Configuration
     4.2.1 Location of UPD Configuration File
     4.2.2 Where Products Get Declared
     4.2.3 Where Products Get Installed
   4.3 Troubleshooting UPD Installation
     4.3.1 What File Permissions Get Set?
     4.3.2 You're Ready to Install: Should you Declare Qualifiers?
     4.3.3 What if an Install Gets Interrupted?
     4.3.4 What if a Product was Installed under a Different Name?
     4.3.5 Proxying Webserver
   4.4 Post-Installation Procedures
     4.4.1 Configuring a Product

  Chapter 4: Product Installation Basics

  This chapter provides general information needed to install products.

  4.1 Installation Methods for UPS Products

  There are three ways to access products from a UPS product distribution node: using UPD , FTP or tar . Each method is described briefly below, and then in more detail in the following chapters. Information on troubleshooting a problematic product installation is provided in Chapter 10: Troubleshooting UPS Product Installations .

  4.1.1 tar

  For UPS product areas with unified layout, it is possible to transfer products from one products area to another by use of the UNIX tar command. The UNIX command tar is used to combine files into a single archive file, for easy storage and distribution. For details, see 7. Installing Products Using tar

To use the tar command, the directory structure of the source and destination product areas must be the same. To check this in ups v5_1 or later, run:

% ups layout

If it returns 'unified', then tar can be used. In the following output:

"/scratch/mengel/p": "unified" 
"/fnal/ups/db": "traditional" 
"/afs/fnal.gov/ups/db": "traditional" 

tar can be used for /scratch/mengel/p.
p<>. For earlier versions of ups , see Checking Layout Before ups V5_1

Assuming the product areas have a unified layout, multiple files and/or directories can be combined into a single file via:

% cd /products/area
% tar czvf /tmp/file.tgz  product1 product2/vx_y_z*

This command packs up all versions of product1 and version vx_y_z of product2 into /tmp/file.tgz. Any name can be used instead of file.tgz, although the .tgz extension should be kept. The c creates a new archive, the z option tells tar to zip the archive as it is created. The v option, verbose, reports all files as they are added. The f option is immediately followed by the archive file name, /tmp/file.tgz in the above example.
p<>. After being created, the archive file is moved to the machine it is to be installed on. This can be via thumb-drive, a shared drive, or however the user wishes.

To extract the files in an archive, enter:

% cd /other/products/area
% tar xzvf /tmp/file.tgz

This will extract the files from the archive, unzip them, and list the files. Chapter 7: Installing Products Using tar provides further details.

  4.1.2 UPD

  The UPD product includes the upd install command for installing products. This is a widely-used product installation method on machines running UPS/UPD . Chapter 6: Installing Products Using UPD is dedicated to describing this process. Installation parameters are set in the local node's UPD configuration. The UPD configuration is described in detail in Chapter 32: The UPD Configuration File .

  The upd install command performs the following functions:

  • retrieves the specified product instance, and by default its dependencies, from a distribution node
  • unwinds the product (if transferred in tar format) and installs it, and by default its dependencies, on the user node according to the node's UPD configuration
  • declares the product, and by default its dependencies, to the database specified in the node's UPD configuration
  • either resolves dependencies or prints to screen the commands needed.

  4.1.3 Installing UPD Products by Hand

  Anonymous FTP is available on fnkits , and may be available on other UPS product distribution nodes. FTP can be used only to retrieve products; it is left to the installer to unwind and declare them.

Unlike the tarfiles mentioned in the previous section, the tarfiles from fnkits do not have the leading product/version/flavor directories or product declaration files; so you have to make those intervening directories, and do a ups declare by hand. This allows UPD to install into multiple/different package layouts, but it is much less convenient for installing by hand.

If the table file and/or the ups directory is (are) not included in the tar file, it (they) must be retrieved separately. Chapter 8: Installing Products using FTP on a UPD Distribution Node describes using FTP to install products.

  FTP is suited to product installations into non-standard locations on your node, e.g., into your own area for use just by you.

  4.2 Modifying Your UPD Configuration

  There are a variety of reasons to modify your UPD configuration:

  • Disk is full, so you want to change the destination of where the products get installed.
  • To separate products so new users only see certain products, but more advanced users have access to other products.
  • If you have a log or mailing list that needs to get updated every time a product is installed.
  • If you need to change permissions whenever someone installs a product.
  • In general, if you have a pre-install or post-install action, you can put that in your UPD configuration file and have it done every time automatically.

When you install a product using UPD , the installation parameters are controlled by the UPD configuration. The UPS configuration file for the database you're using points to a UPD configuration file. These configuration files described in Chapter 31: The UPS Configuration File and Chapter 32: The UPD Configuration File . The UPD configuration file typically consists of one or more stanzas, each of which:

  • identifies certain product instances, products or groups of products
  • specifies a database on the local system in which to declare a product matching the identifier
  • specifies locations on the local system in which UPD is to put a matched product and its related files
  • (optionally) lists actions for UPS/UPD to perform either just before or just after declaring the product

  4.2.1 Location of UPD Configuration File

  The Default UPD Configuration File

  The UPD configuration file is stored as:

  ${UPD_USERCODE_DIR}/updconfig

  where the keyword UPD_USERCODE_DIR is set in the UPS configuration file. It tells the location of the database containing the UPD configuration file. When UPD gets set up, the read-only variable ${UPD_USERCODE_DIR} gets defined and set to the same value as the keyword. (The read-only variable ${UPD_USERCODE_DB} also gets defined and set to the database directory containing ${UPD_USERCODE_DIR}). To find the value of UPD_USERCODE_DIR, you can list the UPS configuration file, e.g.,:

% less $PRODUCTS/.upsfiles/dbconfig

  or you can first setup UPD , and request the variable value, e.g.,:

% echo $UPD_USERCODE_DIR

  or

% env | grep UPD

  Overriding the Default UPD Configuration

  If your system is set up with multiple UPS databases configured to point to different UPD configurations, you can choose to specify a database on the upd install command line pointing to a UPD configuration file other than the default. First, verify that the database you specify points to the UPD configuration you want. To find out, run the command:

% ups list -z <database> -K UPD_USERCODE_DIR

  Note that if this command returns empty quotes, it means the database specifies no configuration file. In this case the default UPD configuration will not be overridden.

  4.2.2 Where Products Get Declared

  The keyword UPS_THIS_DB, set in the UPD configuration file, identifies the database into which UPS declares the product (i.e., the directory that UPD specifies in the ups declare -z <database> option). This keyword may be set differently in different stanzas, thereby causing different products to be declared in different databases.

  4.2.3 Where Products Get Installed

  For organizational reasons it is usually preferable to have UPD configured to install all the UPS products for a database in one area. In the UPS configuration file, typically the keyword PROD_DIR_PREFIX gets set to the product root directory prefix under which the products reside. The UPD configuration file then defines product root directory locations in terms of PROD_DIR_PREFIX. The quantities you need to be aware of within the UPD configuration file are:

  UPS_PROD_DIR

  The product root directory. The upd install command runs the ups declare command, and uses this value as the argument to the -r option. It is usually defined relative to PROD_DIR_PREFIX.

  UNWIND_PROD_DIR

  The absolute path to the directory where products get unwound is usually ${PROD_DIR_PREFIX}/${UPS_PROD_DIR} , however in AFS and some NFS mounting configurations, products are often unwound and declared in different locations.

  UPS_PROD_DIR

  The product root directory. The upd install command runs the ups declare command, and uses this value as the argument to the -r option. It is usually defined relative to PROD_DIR_PREFIX.

  UNWIND_PROD_DIR

  The absolute path to the directory where products get unwound is usually ${PROD_DIR_PREFIX}/${UPS_PROD_DIR} , however in AFS and some NFS mounting configurations, products are often unwound and declared in different locations.

  You should not specify the product location in the upd install command unless you want to override the default.

  4.3 Troubleshooting UPD Installation

  4.3.1 What File Permissions Get Set?

  Product files get downloaded and installed with the same permissions that they have on the distribution node, minus the umask set in your login files. We recommend that you set your umask to 002 before installing any products to ensure that you don't remove the group write access for table files.

  4.3.2 You're Ready to Install: Should you Declare Qualifiers?

  If a product instance is declared with one or more qualifiers on the distribution node, you can choose whether you want to declare it on your system with or without them. Most experiment framework software wants to keep the qualifiers.

If qualifiers are used, users must enter the qualifiers on the command line exactly as they appear in the product declaration each time they want to setup that product instance or perform other UPS operations on it. The files that UPS uses to manage each product allow comment lines (see section 28.1 in Chapter 28: Information Storage Format in Database and Configuration Files ); this provides a way of recording qualifier information if you choose not to declare the qualifiers explicitly.

  4.3.3 What if an Install Gets Interrupted?

  Normally UPD deletes the installed portions of a product when an installation process gets interrupted, and it doesn't declare the pieces that failed to install. Therefore, you generally don't need to worry about cleaning up before reattempting the installation. Just issue the install command again, the same way as you did the first time.

  However, if you interrupted the process for some reason (e.g., you saw it was running out of space), then you'll need to remove by hand the piece that was being installed at the time of the interruption. How will you know? Reattempt the install, and if you get a message similar to this:

directory /a/b/c already exists, will not overwrite.

  then you'll need to remove the specified directory/file(s).

  4.3.4 What if a Product was Installed under a Different Name?

  Giving a product a new name upon installation can cause problems in dependency trees. This practice is not supported, and is certainly not encouraged, but it can be made to work. If you have a product that needs to find, for example, $MYPROD_DIR, but myprod has been installed on your system with a different name, e.g., fermi_myprod , then you may need to edit the table file (described in Chapter 36: Table Files ). Normally product installers never need to touch the table file, but this is an exception. If the provided table file for myprod was written by a developer who has no knowledge of the name change on your system, the table file probably contains:

  ACTION=SETUP

  prodDir()

  where prodDir() instructs the setup command to set the variable $<PRODUCT>_DIR (see 23.1.4 More Detailed Description under section 23.1 setup ). On your system then, the variable $FERMI_MYPROD_DIR will get set, but $MYPROD_DIR won't. To ensure that you also get the variable $MYPROD_DIR, edit the table file and under ACTION=SETUP add the function:

  envSet(MYPROD_DIR, ${UPS_PROD_DIR})

  4.3.5 Proxying Webserver

  Some off-site locations may impose networking restrictions which can interfere with UPD .
If all web traffic is channeled through a proxying webserver at your site, you need to provide the URL of this server to UPD . Since UPD commands go through a web server, they will fail otherwise (the error message will indicate either "Destination unreachable" or "Timeout"). Look at your web browser configuration to find out what proxy you're using. Set the variable http_proxy (lower case) to the URL of your server, e.g., (for C shell):

% setenv http_proxy "http://some.host.name:8000" 

 

  4.4 Post-Installation Procedures

  The average user can skip this section.

Some products require that you perform supplementary steps during or after the installation process, for example copying files to other locations or creating needed files or directories. The product's INSTALL_NOTE file should contain any instructions for completing the installation. Commonly required actions on the installer's part include configuring the product instance.

  4.4.1 Configuring a Product

  Post-installation procedures that can be completely automated are typically collected together such that the command ups configure executes them. This command gets executed by default, as necessary, when the product instance is declared. Otherwise, you can run ups configure manually at any time after declaration to configure the product instance.

  The configuration may involve creating links to the product root directory from other areas (see section 9.1 Installing Products that Require Special Privileges ). If the area is not identical for each node (i.e., same path but separate areas) accessing the UPS database in which the product instance has been declared, then you will need to run the ups configure command manually on each node that mounts a unique area. If you are not sure whether you need to configure a product instance on each node, look through the configuration steps in the table file under ACTION=CONFIGURE to see what they do.

 

TOC PREV NEXT

This page last revised in May 2014

TOC PREV NEXT
Fermilab CD logo Complete Guide and Reference Manual for UPS and UPD

Chapter Contents

Chapter 4: Product Installation Basics
   4.1 Installation Methods for UPS Products
     4.1.1 tar
     4.1.2 UPD
     4.1.3 Installing UPD Products by Hand
   4.2 Modifying Your UPD Configuration
     4.2.1 Location of UPD Configuration File
     4.2.2 Where Products Get Declared
     4.2.3 Where Products Get Installed
   4.3 Troubleshooting UPD Installation
     4.3.1 What File Permissions Get Set?
     4.3.2 You're Ready to Install: Should you Declare Qualifiers?
     4.3.3 What if an Install Gets Interrupted?
     4.3.4 What if a Product was Installed under a Different Name?
     4.3.5 Proxying Webserver
   4.4 Post-Installation Procedures
     4.4.1 Configuring a Product

  Chapter 4: Product Installation Basics

  This chapter provides general information needed to install products.

  4.1 Installation Methods for UPS Products

  There are three ways to access products from a UPS product distribution node: using UPD , FTP or tar . Each method is described briefly below, and then in more detail in the following chapters. Information on troubleshooting a problematic product installation is provided in Chapter 10: Troubleshooting UPS Product Installations .

  4.1.1 tar

  For UPS product areas with unified layout, it is possible to transfer products from one products area to another by use of the UNIX tar command. The UNIX command tar is used to combine files into a single archive file, for easy storage and distribution. For details, see 7. Installing Products Using tar

To use the tar command, the directory structure of the source and destination product areas must be the same. To check this in ups v5_1 or later, run:

% ups layout

If it returns 'unified', then tar can be used. In the following output:

"/scratch/mengel/p": "unified" 
"/fnal/ups/db": "traditional" 
"/afs/fnal.gov/ups/db": "traditional" 

tar can be used for /scratch/mengel/p.
p<>. For earlier versions of ups , see Checking Layout Before ups V5_1

Assuming the product areas have a unified layout, multiple files and/or directories can be combined into a single file via:

% cd /products/area
% tar czvf /tmp/file.tgz  product1 product2/vx_y_z*

This command packs up all versions of product1 and version vx_y_z of product2 into /tmp/file.tgz. Any name can be used instead of file.tgz, although the .tgz extension should be kept. The c creates a new archive, the z option tells tar to zip the archive as it is created. The v option, verbose, reports all files as they are added. The f option is immediately followed by the archive file name, /tmp/file.tgz in the above example.
p<>. After being created, the archive file is moved to the machine it is to be installed on. This can be via thumb-drive, a shared drive, or however the user wishes.

To extract the files in an archive, enter:

% cd /other/products/area
% tar xzvf /tmp/file.tgz

This will extract the files from the archive, unzip them, and list the files. Chapter 7: Installing Products Using tar provides further details.

  4.1.2 UPD

  The UPD product includes the upd install command for installing products. This is a widely-used product installation method on machines running UPS/UPD . Chapter 6: Installing Products Using UPD is dedicated to describing this process. Installation parameters are set in the local node's UPD configuration. The UPD configuration is described in detail in Chapter 32: The UPD Configuration File .

  The upd install command performs the following functions:

  • retrieves the specified product instance, and by default its dependencies, from a distribution node
  • unwinds the product (if transferred in tar format) and installs it, and by default its dependencies, on the user node according to the node's UPD configuration
  • declares the product, and by default its dependencies, to the database specified in the node's UPD configuration
  • either resolves dependencies or prints to screen the commands needed.

  4.1.3 Installing UPD Products by Hand

  Anonymous FTP is available on fnkits , and may be available on other UPS product distribution nodes. FTP can be used only to retrieve products; it is left to the installer to unwind and declare them.

Unlike the tarfiles mentioned in the previous section, the tarfiles from fnkits do not have the leading product/version/flavor directories or product declaration files; so you have to make those intervening directories, and do a ups declare by hand. This allows UPD to install into multiple/different package layouts, but it is much less convenient for installing by hand.

If the table file and/or the ups directory is (are) not included in the tar file, it (they) must be retrieved separately. Chapter 8: Installing Products using FTP on a UPD Distribution Node describes using FTP to install products.

  FTP is suited to product installations into non-standard locations on your node, e.g., into your own area for use just by you.

  4.2 Modifying Your UPD Configuration

  There are a variety of reasons to modify your UPD configuration:

  • Disk is full, so you want to change the destination of where the products get installed.
  • To separate products so new users only see certain products, but more advanced users have access to other products.
  • If you have a log or mailing list that needs to get updated every time a product is installed.
  • If you need to change permissions whenever someone installs a product.
  • In general, if you have a pre-install or post-install action, you can put that in your UPD configuration file and have it done every time automatically.

When you install a product using UPD , the installation parameters are controlled by the UPD configuration. The UPS configuration file for the database you're using points to a UPD configuration file. These configuration files described in Chapter 31: The UPS Configuration File and Chapter 32: The UPD Configuration File . The UPD configuration file typically consists of one or more stanzas, each of which:

  • identifies certain product instances, products or groups of products
  • specifies a database on the local system in which to declare a product matching the identifier
  • specifies locations on the local system in which UPD is to put a matched product and its related files
  • (optionally) lists actions for UPS/UPD to perform either just before or just after declaring the product

  4.2.1 Location of UPD Configuration File

  The Default UPD Configuration File

  The UPD configuration file is stored as:

  ${UPD_USERCODE_DIR}/updconfig

  where the keyword UPD_USERCODE_DIR is set in the UPS configuration file. It tells the location of the database containing the UPD configuration file. When UPD gets set up, the read-only variable ${UPD_USERCODE_DIR} gets defined and set to the same value as the keyword. (The read-only variable ${UPD_USERCODE_DB} also gets defined and set to the database directory containing ${UPD_USERCODE_DIR}). To find the value of UPD_USERCODE_DIR, you can list the UPS configuration file, e.g.,:

% less $PRODUCTS/.upsfiles/dbconfig

  or you can first setup UPD , and request the variable value, e.g.,:

% echo $UPD_USERCODE_DIR

  or

% env | grep UPD

  Overriding the Default UPD Configuration

  If your system is set up with multiple UPS databases configured to point to different UPD configurations, you can choose to specify a database on the upd install command line pointing to a UPD configuration file other than the default. First, verify that the database you specify points to the UPD configuration you want. To find out, run the command:

% ups list -z <database> -K UPD_USERCODE_DIR

  Note that if this command returns empty quotes, it means the database specifies no configuration file. In this case the default UPD configuration will not be overridden.

  4.2.2 Where Products Get Declared

  The keyword UPS_THIS_DB, set in the UPD configuration file, identifies the database into which UPS declares the product (i.e., the directory that UPD specifies in the ups declare -z <database> option). This keyword may be set differently in different stanzas, thereby causing different products to be declared in different databases.

  4.2.3 Where Products Get Installed

  For organizational reasons it is usually preferable to have UPD configured to install all the UPS products for a database in one area. In the UPS configuration file, typically the keyword PROD_DIR_PREFIX gets set to the product root directory prefix under which the products reside. The UPD configuration file then defines product root directory locations in terms of PROD_DIR_PREFIX. The quantities you need to be aware of within the UPD configuration file are:

  UPS_PROD_DIR

  The product root directory. The upd install command runs the ups declare command, and uses this value as the argument to the -r option. It is usually defined relative to PROD_DIR_PREFIX.

  UNWIND_PROD_DIR

  The absolute path to the directory where products get unwound is usually ${PROD_DIR_PREFIX}/${UPS_PROD_DIR} , however in AFS and some NFS mounting configurations, products are often unwound and declared in different locations.

  UPS_PROD_DIR

  The product root directory. The upd install command runs the ups declare command, and uses this value as the argument to the -r option. It is usually defined relative to PROD_DIR_PREFIX.

  UNWIND_PROD_DIR

  The absolute path to the directory where products get unwound is usually ${PROD_DIR_PREFIX}/${UPS_PROD_DIR} , however in AFS and some NFS mounting configurations, products are often unwound and declared in different locations.

  You should not specify the product location in the upd install command unless you want to override the default.

  4.3 Troubleshooting UPD Installation

  4.3.1 What File Permissions Get Set?

  Product files get downloaded and installed with the same permissions that they have on the distribution node, minus the umask set in your login files. We recommend that you set your umask to 002 before installing any products to ensure that you don't remove the group write access for table files.

  4.3.2 You're Ready to Install: Should you Declare Qualifiers?

  If a product instance is declared with one or more qualifiers on the distribution node, you can choose whether you want to declare it on your system with or without them. Most experiment framework software wants to keep the qualifiers.

If qualifiers are used, users must enter the qualifiers on the command line exactly as they appear in the product declaration each time they want to setup that product instance or perform other UPS operations on it. The files that UPS uses to manage each product allow comment lines (see section 28.1 in Chapter 28: Information Storage Format in Database and Configuration Files ); this provides a way of recording qualifier information if you choose not to declare the qualifiers explicitly.

  4.3.3 What if an Install Gets Interrupted?

  Normally UPD deletes the installed portions of a product when an installation process gets interrupted, and it doesn't declare the pieces that failed to install. Therefore, you generally don't need to worry about cleaning up before reattempting the installation. Just issue the install command again, the same way as you did the first time.

  However, if you interrupted the process for some reason (e.g., you saw it was running out of space), then you'll need to remove by hand the piece that was being installed at the time of the interruption. How will you know? Reattempt the install, and if you get a message similar to this:

directory /a/b/c already exists, will not overwrite.

  then you'll need to remove the specified directory/file(s).

  4.3.4 What if a Product was Installed under a Different Name?

  Giving a product a new name upon installation can cause problems in dependency trees. This practice is not supported, and is certainly not encouraged, but it can be made to work. If you have a product that needs to find, for example, $MYPROD_DIR, but myprod has been installed on your system with a different name, e.g., fermi_myprod , then you may need to edit the table file (described in Chapter 36: Table Files ). Normally product installers never need to touch the table file, but this is an exception. If the provided table file for myprod was written by a developer who has no knowledge of the name change on your system, the table file probably contains:

  ACTION=SETUP

  prodDir()

  where prodDir() instructs the setup command to set the variable $<PRODUCT>_DIR (see 23.1.4 More Detailed Description under section 23.1 setup ). On your system then, the variable $FERMI_MYPROD_DIR will get set, but $MYPROD_DIR won't. To ensure that you also get the variable $MYPROD_DIR, edit the table file and under ACTION=SETUP add the function:

  envSet(MYPROD_DIR, ${UPS_PROD_DIR})

  4.3.5 Proxying Webserver

  Some off-site locations may impose networking restrictions which can interfere with UPD .
If all web traffic is channeled through a proxying webserver at your site, you need to provide the URL of this server to UPD . Since UPD commands go through a web server, they will fail otherwise (the error message will indicate either "Destination unreachable" or "Timeout"). Look at your web browser configuration to find out what proxy you're using. Set the variable http_proxy (lower case) to the URL of your server, e.g., (for C shell):

% setenv http_proxy "http://some.host.name:8000" 

 

  4.4 Post-Installation Procedures

  The average user can skip this section.

Some products require that you perform supplementary steps during or after the installation process, for example copying files to other locations or creating needed files or directories. The product's INSTALL_NOTE file should contain any instructions for completing the installation. Commonly required actions on the installer's part include configuring the product instance.

  4.4.1 Configuring a Product

  Post-installation procedures that can be completely automated are typically collected together such that the command ups configure executes them. This command gets executed by default, as necessary, when the product instance is declared. Otherwise, you can run ups configure manually at any time after declaration to configure the product instance.

  The configuration may involve creating links to the product root directory from other areas (see section 9.1 Installing Products that Require Special Privileges ). If the area is not identical for each node (i.e., same path but separate areas) accessing the UPS database in which the product instance has been declared, then you will need to run the ups configure command manually on each node that mounts a unique area. If you are not sure whether you need to configure a product instance on each node, look through the configuration steps in the table file under ACTION=CONFIGURE to see what they do.

 

TOC PREV NEXT

This page last revised in May 2014