Project

General

Profile

TOC PREV NEXT
Fermilab CD logo Complete Guide and Reference Manual for UPS and UPD

Chapter Contents

Chapter 28: Information Storage Format in Database and Configuration Files
   28.1 Overview of File Types
   28.2 Keywords: Information Storage Format
     28.2.1 What is a Keyword?
     28.2.2 Keyword Syntax
     28.2.3 User-Defined Keywords
     28.2.4 How UPS/UPD Sets Keyword Values
   28.3 Flexibility of File Syntax
   28.4 List of Supported Keywords
   28.5 Syntax for Assigning Keyword Values
   28.6 Usage Notes on Particular Keywords
     28.6.1 COMPILE_DIR, COMPILE_FILE and @COMPILE_FILE
     28.6.2 PROD_DIR_PREFIX, PROD_DIR and @PROD_DIR
     28.6.3 STATISTICS
     28.6.4 TABLE_FILE and @TABLE_FILE
     28.6.5 UPS_DIR and @UPS_DIR
     28.6.6 _UPD_OVERLAY

  Chapter 28: Information Storage Format in Database and Configuration Files

  Most of the time, product installers and UPS database managers can get all the information they need about a product or about the contents of a database via the ups list [-K <keywordList>] command output (described in section 23.11 ups list ), which is fairly easy to interpret. However, it's helpful to understand the database files when dealing with complex situations. The keywords described in this chapter which appear in the database files also appear in the ups list -l output.

  28.1 [[files_intro#34874|]] Overview of File Types

  The information that UPS needs to configure and manage a database and to identify, locate, and retrieve product instances resides in a set of ASCII files in the UPS database. The information that UPD needs for installing products also resides there. The files used for these purposes include:

  • Version files tell UPS where to find all the files associated with a particular version of a product on the local system. They contain information specific to the local installation of the product. They are generally named according to the scheme vx_y.version , e.g., v1_0.version . These are described in Chapter 29: Version Files .
  • Chain files are optional and contain pointers to version files, thus providing convenient access to particular product versions on the local system. They are generally named according to the scheme chainname.chain . These are described in Chapter 30: Chain Files .
  • The UPS database configuration file defines which nodes can access products maintained in the database, and which directories house products, man pages, UPS initialization files, the UPD configuration file, and so on. It is described in Chapter 31: The UPS Configuration File .
  • The UPD configuration file controls where UPD installs products and miscellaneous product-related files. It can also be used to define supplementary actions for UPD to perform when installing or updating products. It is described in Chapter 32: The UPD Configuration File .

  These files are sometimes referred to collectively as UPS database files . They store information in the format of keywords .

  This information storage format is also used in table files , which are provided by the product developer and discussed in Part VIII Developer's Reference . They contain product-specific, system-independent information. Table files can be maintained in the database, but they are not constrained to reside there, and in fact usually reside under the product root directory.

  28.2 Keywords: Information Storage Format

  28.2.1 What is a Keyword?

  UPS/UPD utilizes a set of keywords that collectively store the information UPS/UPD requires for managing products. A keyword represents a category of information used by UPS/UPD , it is akin to a variable. A keyword line in a file assigns a value to a keyword in the format KEYWORD = VALUE .

  The supported keywords are listed and described in the table in section 28.4 List of Supported Keywords . Some of the keywords can be used in all the file types, others are restricted to certain file types. A few keywords have default values.

  Keywords and their values are not case-sensitive.

  28.2.2 Keyword Syntax

  When two or more words are used to make up one keyword, they are generally separated by an underscore (_) for readability. All the provided keywords use full words except:

  DB

  is used instead of DATABASE

  DIR

  is used instead of DIRECTORY

  PROD

  is used instead of PRODUCT

  28.2.3 User-Defined Keywords

  In addition to those listed, UPS/UPD allows user-defined keywords (where user in this context refers to a product developer or administrative user). All user-defined keywords must have underscore ( _ ) as the initial character. While parsing, any unrecognized (i.e., user-defined) keywords are ignored by UPS , but they are preserved across rewrites of the files.

  28.2.4 How UPS/UPD Sets Keyword Values

  Keywords stored in the UPS database configuration file (described in Chapter 31: The UPS Configuration File ) and the UPD configuration file (described in Chapter 32: The UPD Configuration File ) are given values according to the configuration chosen when UPS/UPD was installed and configured. See Chapter 14: Installing UPS and UPD from Bootstrap for information on choosing values during the installation of UPS/UPD .

  Keywords stored in version or chain files are set at the time that the corresponding product instance and/or chain is declared to the UPS database. Those stored in table files are usually set by the product developer. If a keyword is stored in both the database configuration file and another file, then, for the corresponding product instance(s), the value set at product or chain declaration overrides the one set in the database configuration file.

  28.3 Flexibility of File Syntax

  The syntax of the database files is fixed but forgiving . It is fixed in the sense that UPS commands automatically create the version and chain files in a particular UPS -supported format. Any UPS command that modifies information in these files rewrites the file to disk according to the same format. The syntax is forgiving , however, in that when you perform manual file updates, UPS will ignore blank lines and extra whitespace (spaces and tabs).

  Comment lines can be placed anywhere in the file and must begin with a pound sign ( # ). However, if you want comments to be preserved upon rewrite, they must be the first lines in the file.

  28.4 List of Supported Keywords

  The following table gives information about each provided keyword. The last five columns indicate which database file the keyword may be used in. The headings D, U, C, V and T refer to:
DDatabase configuration file ( dbconfig )U UPD configuration file ( updconfig )CChain fileVVersion fileTTable file
p<>.  

Keyword and Keyword and Default Value (if any) Default Value (if any) Description and Description and Notes (if any) Notes (if any) D D U U C C V V T T
ACTION ACTION defines an action (described in Chapter 34: Actions and ACTION Keyword Values ), i.e., groups together a list of functions associated with a command (e.g., ACTION=SETUP) defines an action (described in Chapter 34: Actions and ACTION Keyword Values ), i.e., groups together a list of functions associated with a command (e.g., ACTION=SETUP)
ARCHIVE_FILE ARCHIVE_FILE archive file name/location; used by UPD archive file name/location; used by UPD
AUTHORIZED_NODES AUTHORIZED_NODES Default: All nodes (); taken from UPS database configuration file Default: All nodes (); taken from UPS database configuration file authorized nodes authorized nodes D D
CHAIN CHAIN chain name chain name
COMMON: COMMON: groups together actions that apply to all instances represented in "GROUP:"; groups together actions that apply to all instances represented in "GROUP:"; COMMON: is only valid within a GROUP: COMMON: is only valid within a GROUP:
COMPILE_DIR COMPILE_DIR directory in which the compile file resides directory in which the compile file resides
COMPILE_FILE COMPILE_FILE the name of the file containing compiled functions (see Chapter 38: Use of Compile Scripts in Table Files ) the name of the file containing compiled functions (see Chapter 38: Use of Compile Scripts in Table Files )
DECLARED DECLARED Default: current date and time Default: current date and time the date/time that the instance was declared to UPS or declared with a chain the date/time that the instance was declared to UPS or declared with a chain Note: often has multiple values, one for each declaration (e.g., for subsequent chain declarations) Note: often has multiple values, one for each declaration (e.g., for subsequent chain declarations)
DECLARER DECLARER Default: current user Default: current user userid of user that performed the declaration userid of user that performed the declaration Note: often has multiple values, one for each declaration (e.g., for subsequent chain declarations) Note: often has multiple values, one for each declaration (e.g., for subsequent chain declarations)
DESCRIPTION DESCRIPTION product description product description
END: END: marks the end of a "GROUP:" or "COMMON:"; one "END:" marker is used to jointly end a "GROUP:" and an included "COMMON:" marks the end of a "GROUP:" or "COMMON:"; one "END:" marker is used to jointly end a "GROUP:" and an included "COMMON:"
FILE FILE type of file (possible values: DBCONFIG, UPDCONFIG, CHAIN, VERSION, TABLE) type of file (possible values: DBCONFIG, UPDCONFIG, CHAIN, VERSION, TABLE) D D U U C C V V T T
FLAVOR FLAVOR product instance flavor product instance flavor Note: To easily accommodate flavor-neutral setup functions in a table file, FLAVOR can take the value ANY, but only in a table file. Note: To easily accommodate flavor-neutral setup functions in a table file, FLAVOR can take the value ANY, but only in a table file.
GROUP: GROUP: groups together multiple instances; all entries subsequent to this "GROUP:" are part of it until an "END:" marker is reached groups together multiple instances; all entries subsequent to this "GROUP:" are part of it until an "END:" marker is reached
INFO_SOURCE_DIR INFO_SOURCE_DIR Default: under the ${UPS_UPS_DIR}/ toInfo directory Default: under the ${UPS_UPS_DIR}/ toInfo directory location of Info files included with instance location of Info files included with instance
INFO_TARGET_DIR INFO_TARGET_DIR directory into which Info files are to be copied directory into which Info files are to be copied D D
MAN_SOURCE_DIR MAN_SOURCE_DIR Default: under the ${UPS_UPS_DIR}/ toman/man directory Default: under the ${UPS_UPS_DIR}/ toman/man directory location of unformatted man page files included with instance location of unformatted man page files included with instance
MAN_TARGET_DIR MAN_TARGET_DIR directory into which formatted man pages are to be copied directory into which formatted man pages are to be copied D D
MODIFIED MODIFIED Default: Current date/time Default: Current date/time last time the associated instance was changed last time the associated instance was changed Note: often has multiple values, one for each declaration/modification (e.g., for subsequent chain declarations) Note: often has multiple values, one for each declaration/modification (e.g., for subsequent chain declarations)
MODIFIER MODIFIER Default: Current user Default: Current user userid of user that modified the instance userid of user that modified the instance Note: often has multiple values, one for each declaration/modification (e.g., for subsequent chain declarations) Note: often has multiple values, one for each declaration/modification (e.g., for subsequent chain declarations)
ORIGIN ORIGIN master source file; see option -D in Chapter 25: Generic Command Option Descriptions master source file; see option -D in Chapter 25: Generic Command Option Descriptions
PRODUCT PRODUCT product name product name
PROD_DIR PROD_DIR product root directory (usually defined relative to PROD_DIR_PREFIX, below) product root directory (usually defined relative to PROD_DIR_PREFIX, below)
PROD_DIR_PREFIX PROD_DIR_PREFIX product root directory prefix (area where all or most product instances are maintained) product root directory prefix (area where all or most product instances are maintained) D D
QUALIFIERS QUALIFIERS additional instance specification information often used to indicate compilation options used by developer additional instance specification information often used to indicate compilation options used by developer Notes: appears immediately after a FLAVOR in these files, and is coupled with it to complete the instance identification (see 27.2.3 Qualifiers: Use in Instance Matching ) Notes: appears immediately after a FLAVOR in these files, and is coupled with it to complete the instance identification (see 27.2.3 Qualifiers: Use in Instance Matching )
SETUPS_DIR SETUPS_DIR location of setups.[c]sh files and other UPS initialization files location of setups.[c]sh files and other UPS initialization files D D
STATISTICS STATISTICS flag to record statistics for specified products flag to record statistics for specified products See 28.6.3 STATISTICS for usage information. See 28.6.3 STATISTICS for usage information. D D
TABLE_DIR TABLE_DIR Default: search path (see section 29.4 Determination of ups Directory and Table File Locations ) Default: search path (see section 29.4 Determination of ups Directory and Table File Locations ) location of table file location of table file
TABLE_FILE TABLE_FILE name of table file (relative to TABLE_DIR) name of table file (relative to TABLE_DIR)
UNWIND_ARCHIVE_ FILE UNWIND_ARCHIVE_ FILE (a UPD keyword used only on distribution server configurations) (a UPD keyword used only on distribution server configurations) absolute path to directory in which to unwind archive file (tar file) of product absolute path to directory in which to unwind archive file (tar file) of product
UNWIND_PROD_DIR UNWIND_PROD_DIR (a UPD keyword) absolute path to directory where product gets unwound (a UPD keyword) absolute path to directory where product gets unwound
UNWIND_TABLE_DIR UNWIND_TABLE_DIR (a UPD keyword) absolute path to directory where the table file gets unwound (a UPD keyword) absolute path to directory where the table file gets unwound
UNWIND_UPS_DIR UNWIND_UPS_DIR (a UPD keyword) absolute path to directory where the ups directory gets unwound (a UPD keyword) absolute path to directory where the ups directory gets unwound
UPD_USERCODE_DB UPD_USERCODE_DB Database containing UPD_USERCODE_DIR (set internally) Database containing UPD_USERCODE_DIR (set internally)
UPD_USERCODE_DIR UPD_USERCODE_DIR Directory where UPD configuration files are maintained Directory where UPD configuration files are maintained D D
UPS_ARCHIVE_FILE UPS_ARCHIVE_FILE (a UPD keyword used only on distribution server configurations) (a UPD keyword used only on distribution server configurations) archive file (tar file) location that UPD specifies in archive file (tar file) location that UPD specifies in ups declare -T ftp://host${UPS_ARCHIVE_FILE} ups declare -T ftp://host${UPS_ARCHIVE_FILE}
UPS_DB_VERSION UPS_DB_VERSION UPS database version UPS database version D D
UPS_DIR UPS_DIR Default: ${UPS_PROD_DIR}/ups if directory exists there Default: ${UPS_PROD_DIR}/ups if directory exists there location of ups directory (if not absolute path, then taken relative to PROD_DIR, if specified) location of ups directory (if not absolute path, then taken relative to PROD_DIR, if specified)
UPS_PROD_DIR UPS_PROD_DIR (a UPD keyword) product root directory that UPD specifies in the ups declare -r option; should be defined relative to PROD_DIR_PREFIX for portability (a UPD keyword) product root directory that UPD specifies in the ups declare -r option; should be defined relative to PROD_DIR_PREFIX for portability
UPS_TABLE_DIR UPS_TABLE_DIR (a UPD keyword) table file directory that UPD specifies in the ups declare -M option (a UPD keyword) table file directory that UPD specifies in the ups declare -M option % %{font-weight: bold;color: #000000}Normally this should not be set ! Since UPS_TABLE_DIR must be an absolute path, the declaration becomes non-portable if you set this location. % %{font-weight: bold;color: #000000}Normally this should not be set ! Since UPS_TABLE_DIR must be an absolute path, the declaration becomes non-portable if you set this location.
UPS_THIS_DB UPS_THIS_DB (a UPD keyword) the database into which UPS declares the product (i.e., the directory that UPD specifies in the ups declare -z option). (a UPD keyword) the database into which UPS declares the product (i.e., the directory that UPD specifies in the ups declare -z option).
UPS_UPS_DIR UPS_UPS_DIR (a UPD keyword) ups directory that UPD specifies in the ups declare -U option, taken relative to ${UNWIND_PROD_DIR} unless an absolute path is given; usually defined as ups. (a UPD keyword) ups directory that UPD specifies in the ups declare -U option, taken relative to ${UNWIND_PROD_DIR} unless an absolute path is given; usually defined as ups.
UPS_TABLE_FILE UPS_TABLE_FILE (a UPD keyword) table file name that UPD specifies in the ups declare -m option (a UPD keyword) table file name that UPD specifies in the ups declare -m option
USER USER current username current username
VERSION VERSION product version product version
_UPD_OVERLAY _UPD_OVERLAY main product name for overlaid product main product name for overlaid product Note: This keyword is user-defined from UPS 's point of view. It is included here because it is configured and used by UPD . Its use with overlaid products is described in section 28.6.6 _UPD_OVERLAY . Note: This keyword is user-defined from UPS 's point of view. It is included here because it is configured and used by UPD . Its use with overlaid products is described in section 28.6.6 _UPD_OVERLAY .
|
|
|
T T

  28.5 Syntax for Assigning Keyword Values

  • Any keyword value that has multiple values uses a colon ( : ) to separate the subvalues. The value (i.e., the list of subvalues) may be surrounded by double quotation marks ( "..." ). Blanks within the double-quoted value are ignored; they are neither required nor prohibited.
    p<>.   For example, the following are all equivalent:

  QUALIFIERS = debug:optimize

  QUALIFIERS = "debug:optimize"

  QUALIFIERS = " debug: optimize"

  • Whitespace is ignored except within the keyword values for DESCRIPTION, DECLARER and MODIFIER
  • Leading whitespace is ignored.
  • There are no line continuation characters; the entire keyword definition or function must appear on a single line.
  • The "at" character (@) is defined for use with the keywords COMPILE_FILE, PROD_DIR, UPS_DIR and TABLE_FILE. See section 28.6 Usage Notes on Particular Keywords .

  28.6 Usage Notes on Particular Keywords

  28.6.1 COMPILE_DIR, COMPILE_FILE and @COMPILE_FILE

  COMPILE_DIR

  the directory in which the compile file resides (see Chapter 38: Use of Compile Scripts in Table Files )

  COMPILE_FILE

  the name of the file containing precompiled functions

  @COMPILE_FILE

  the entire path to the file containing precompiled functions

  28.6.2 PROD_DIR_PREFIX, PROD_DIR and @PROD_DIR

  PROD_DIR_PREFIX

  is generally set to the root of the path shared by all the products.

  PROD_DIR

  is the path that gets specified when the particular product instance is declared; it is usually (but not always) a relative path that gets tacked onto PROD_DIR_PREFIX.

  @PROD_DIR

  is a shorthand to request the entire path for the directory where the product is installed (usually equivalent to PROD_DIR_PREFIX/PROD_DIR).

  If PROD_DIR_PREFIX is not defined on your system, then PROD_DIR should represent the entire path, in which case PROD_DIR and @PROD_DIR are identical.

  Compare these commands and their output:

% ups list -K PROD_DIR_PREFIX sam_docs
"/afs/fnal.gov/ups/prd" 
% ups list -K PROD_DIR sam_docs
"sam_docs/v1_0/NULL" 
% ups list -K @PROD_DIR sam_docs
"/afs/fnal.gov/ups/prd/sam_docs/v1_0/NULL" 

  28.6.3 STATISTICS

  The STATISTICS keyword is provided to allow recording of the following statistics on product usage and UPS database access:

  • Userid of person executing UPS/UPD command
  • Date and time
  • Which command was executed (including options and arguments)
  • Which product instance was selected by command

  This keyword can appear in a product's version file and/or in the UPS database configuration file, thus providing a great deal of flexibility in choosing which products/instances to monitor.

  Use in a Version File

  When the STATISTICS keyword is present in a version file, it must be included with each specific instance which is to be monitored. If the STATISTICS keyword is located before any FLAVOR and/or QUALIFIERS keywords (these keywords separate out different instances), then it is ignored. In a version file, this keyword should have no value assigned.

  Use in a Database Configuration File

  When the STATISTICS keyword appears in the database configuration file, it needs a value. (If it has no value, it is ignored.) Its value is a colon-separated list of the products (name only) on which to record statistics (e.g., STATISTICS = "tcl:tk:cern" ). The value * (asterisk) indicates that statistics are to be gathered on all products in the database.

  Statistics Output

  For a given product being monitored, statistics data for the product get recorded in a file whose name is the same as the product. If the product has dependencies, data also get recorded for them in their own product-specific files, and the data include the parent product name and version number. The data get recorded only when the UPS/UPD command in question has succeeded (i.e., when the temporary file has been created, but not yet sourced).

  The statistics output files for all the monitored products and their dependencies reside in a special directory associated with the UPS database, namely $PRODUCTS/.upsfiles/statistics . This makes it easy to determine which products are being monitored, and only one directory needs to be made world-writable.

  As an example of the statistics data that get recorded, let's look at the tcl product. It is a dependency of tk . Data that are recorded when an instance of tcl is accessed independently look like this:

"tcl" "v8_0" "Linux" "" "" "user1" "2013-03-18 15.22.36 GMT" "setup" 

  Data that are recorded for tcl when an instance of tk is accessed look like this:

"tcl" "v8_0" "Linux" "" "" "user1" "2013-03-18 15.22.36 GMT" "setupRequired tk v8_0" 

  Statistics Example

  The system can be set up to log a subset of information, such as the version of the software and the top level product, instead of everything. Once the data has been collected, users can grab parts of the output file for specific information as shown in the following example:

set "STATISTICS = larsoft:lbne_software" and run commands.

Then

% cd /path/to/db/.upsfiles/statistics

% cat * | cut -d '"' -f 12  | sort | uniq -c
to get a count of how many times each user set things up

The above command tells the 'cut' command to use a double quote as a separator. Ups lists things in the log file like this: "prod" "vers" "flavor" "quals" "" "user" "date" "command"
p<>. The 'cut' command, when it uses a double quote as a separator, counts the whitespace (and the empty string preceding the first double quote) as fields, too. So to get the username, which is the 6th item in the file, use:

cut -d '"' -f 12
To get the hour of the day, we use one cut to extract the date, 
   cut -d '"' -f 14
     and then use another cut command to pull the hour out by column number
   cut -c 12-13

    We then run that text into
     sort | uniq -c
    to give a count of how many times each item occurs.

So, to put it all together, use:
% cat * | cut -d '"' -f 14  | cut -c 12-13 | sort | uniq -c
to get count by hour of the day

  28.6.4 TABLE_FILE and @TABLE_FILE

  TABLE_FILE represents only the name of the table file, not its path. @TABLE_FILE is the entire path for the table file. Compare these commands and their output:

% ups list -Ktable_file sam_docs
"v1_0.table" 
% ups list -K@table_file sam_docs
"/afs/fnal.gov/ups/db/sam_docs/v1_0.table" 

  See section 29.4 Determination of ups Directory and Table File Locations for information on how UPS determines the table file directory.

  28.6.5 UPS_DIR and @UPS_DIR

  UPS_DIR represents the location of the product's ups directory. If it is not an absolute path, then it is taken relative to @PROD_DIR (as shown in the example below). @UPS_DIR is the absolute path. Compare these commands and their output:

% ups list -K @PROD_DIR sam_docs
"/afs/fnal.gov/ups/prd/sam_docs/v1_0/NULL" 
% ups list -Kups_dir sam_docs
"ups" 
% ups list -K@ups_dir sam_docs
"/afs/fnal.gov/ups/prd/sam_docs/v1_0/NULL/ups" 

  28.6.6 _UPD_OVERLAY

  The _UPD_OVERLAY keyword defined in UPD is provided for inclusion in the table file of each overlaid product. Overlaid products are introduced in section 2.3.7 Product Overlays and discussed again for developers in section 17.2.4 Overlaid Products . _UPD_OVERLAY takes as its value the main product name in double quotes. Its presence indicates that the product is an overlaid product maintained in the root directory of the main product listed as the keyword's value. For example, the table files for the products cern_bin , cern_ups , and cern_lib would contain the following keyword line:

  _UPD_OVERLAY = "cern"

  UPD would then use cern as the product name when determining the root directory.

 

TOC PREV NEXT

Last revised in July 2014

TOC PREV NEXT
Fermilab CD logo Complete Guide and Reference Manual for UPS and UPD

Chapter Contents

Chapter 28: Information Storage Format in Database and Configuration Files
   28.1 Overview of File Types
   28.2 Keywords: Information Storage Format
     28.2.1 What is a Keyword?
     28.2.2 Keyword Syntax
     28.2.3 User-Defined Keywords
     28.2.4 How UPS/UPD Sets Keyword Values
   28.3 Flexibility of File Syntax
   28.4 List of Supported Keywords
   28.5 Syntax for Assigning Keyword Values
   28.6 Usage Notes on Particular Keywords
     28.6.1 COMPILE_DIR, COMPILE_FILE and @COMPILE_FILE
     28.6.2 PROD_DIR_PREFIX, PROD_DIR and @PROD_DIR
     28.6.3 STATISTICS
     28.6.4 TABLE_FILE and @TABLE_FILE
     28.6.5 UPS_DIR and @UPS_DIR
     28.6.6 _UPD_OVERLAY

  Chapter 28: Information Storage Format in Database and Configuration Files

  Most of the time, product installers and UPS database managers can get all the information they need about a product or about the contents of a database via the ups list [-K <keywordList>] command output (described in section 23.11 ups list ), which is fairly easy to interpret. However, it's helpful to understand the database files when dealing with complex situations. The keywords described in this chapter which appear in the database files also appear in the ups list -l output.

  28.1 [[files_intro#34874|]] Overview of File Types

  The information that UPS needs to configure and manage a database and to identify, locate, and retrieve product instances resides in a set of ASCII files in the UPS database. The information that UPD needs for installing products also resides there. The files used for these purposes include:

  • Version files tell UPS where to find all the files associated with a particular version of a product on the local system. They contain information specific to the local installation of the product. They are generally named according to the scheme vx_y.version , e.g., v1_0.version . These are described in Chapter 29: Version Files .
  • Chain files are optional and contain pointers to version files, thus providing convenient access to particular product versions on the local system. They are generally named according to the scheme chainname.chain . These are described in Chapter 30: Chain Files .
  • The UPS database configuration file defines which nodes can access products maintained in the database, and which directories house products, man pages, UPS initialization files, the UPD configuration file, and so on. It is described in Chapter 31: The UPS Configuration File .
  • The UPD configuration file controls where UPD installs products and miscellaneous product-related files. It can also be used to define supplementary actions for UPD to perform when installing or updating products. It is described in Chapter 32: The UPD Configuration File .

  These files are sometimes referred to collectively as UPS database files . They store information in the format of keywords .

  This information storage format is also used in table files , which are provided by the product developer and discussed in Part VIII Developer's Reference . They contain product-specific, system-independent information. Table files can be maintained in the database, but they are not constrained to reside there, and in fact usually reside under the product root directory.

  28.2 Keywords: Information Storage Format

  28.2.1 What is a Keyword?

  UPS/UPD utilizes a set of keywords that collectively store the information UPS/UPD requires for managing products. A keyword represents a category of information used by UPS/UPD , it is akin to a variable. A keyword line in a file assigns a value to a keyword in the format KEYWORD = VALUE .

  The supported keywords are listed and described in the table in section 28.4 List of Supported Keywords . Some of the keywords can be used in all the file types, others are restricted to certain file types. A few keywords have default values.

  Keywords and their values are not case-sensitive.

  28.2.2 Keyword Syntax

  When two or more words are used to make up one keyword, they are generally separated by an underscore (_) for readability. All the provided keywords use full words except:

  DB

  is used instead of DATABASE

  DIR

  is used instead of DIRECTORY

  PROD

  is used instead of PRODUCT

  28.2.3 User-Defined Keywords

  In addition to those listed, UPS/UPD allows user-defined keywords (where user in this context refers to a product developer or administrative user). All user-defined keywords must have underscore ( _ ) as the initial character. While parsing, any unrecognized (i.e., user-defined) keywords are ignored by UPS , but they are preserved across rewrites of the files.

  28.2.4 How UPS/UPD Sets Keyword Values

  Keywords stored in the UPS database configuration file (described in Chapter 31: The UPS Configuration File ) and the UPD configuration file (described in Chapter 32: The UPD Configuration File ) are given values according to the configuration chosen when UPS/UPD was installed and configured. See Chapter 14: Installing UPS and UPD from Bootstrap for information on choosing values during the installation of UPS/UPD .

  Keywords stored in version or chain files are set at the time that the corresponding product instance and/or chain is declared to the UPS database. Those stored in table files are usually set by the product developer. If a keyword is stored in both the database configuration file and another file, then, for the corresponding product instance(s), the value set at product or chain declaration overrides the one set in the database configuration file.

  28.3 Flexibility of File Syntax

  The syntax of the database files is fixed but forgiving . It is fixed in the sense that UPS commands automatically create the version and chain files in a particular UPS -supported format. Any UPS command that modifies information in these files rewrites the file to disk according to the same format. The syntax is forgiving , however, in that when you perform manual file updates, UPS will ignore blank lines and extra whitespace (spaces and tabs).

  Comment lines can be placed anywhere in the file and must begin with a pound sign ( # ). However, if you want comments to be preserved upon rewrite, they must be the first lines in the file.

  28.4 List of Supported Keywords

  The following table gives information about each provided keyword. The last five columns indicate which database file the keyword may be used in. The headings D, U, C, V and T refer to:
DDatabase configuration file ( dbconfig )U UPD configuration file ( updconfig )CChain fileVVersion fileTTable file
p<>.  

Keyword and Keyword and Default Value (if any) Default Value (if any) Description and Description and Notes (if any) Notes (if any) D D U U C C V V T T
ACTION ACTION defines an action (described in Chapter 34: Actions and ACTION Keyword Values ), i.e., groups together a list of functions associated with a command (e.g., ACTION=SETUP) defines an action (described in Chapter 34: Actions and ACTION Keyword Values ), i.e., groups together a list of functions associated with a command (e.g., ACTION=SETUP)
ARCHIVE_FILE ARCHIVE_FILE archive file name/location; used by UPD archive file name/location; used by UPD
AUTHORIZED_NODES AUTHORIZED_NODES Default: All nodes (); taken from UPS database configuration file Default: All nodes (); taken from UPS database configuration file authorized nodes authorized nodes D D
CHAIN CHAIN chain name chain name
COMMON: COMMON: groups together actions that apply to all instances represented in "GROUP:"; groups together actions that apply to all instances represented in "GROUP:"; COMMON: is only valid within a GROUP: COMMON: is only valid within a GROUP:
COMPILE_DIR COMPILE_DIR directory in which the compile file resides directory in which the compile file resides
COMPILE_FILE COMPILE_FILE the name of the file containing compiled functions (see Chapter 38: Use of Compile Scripts in Table Files ) the name of the file containing compiled functions (see Chapter 38: Use of Compile Scripts in Table Files )
DECLARED DECLARED Default: current date and time Default: current date and time the date/time that the instance was declared to UPS or declared with a chain the date/time that the instance was declared to UPS or declared with a chain Note: often has multiple values, one for each declaration (e.g., for subsequent chain declarations) Note: often has multiple values, one for each declaration (e.g., for subsequent chain declarations)
DECLARER DECLARER Default: current user Default: current user userid of user that performed the declaration userid of user that performed the declaration Note: often has multiple values, one for each declaration (e.g., for subsequent chain declarations) Note: often has multiple values, one for each declaration (e.g., for subsequent chain declarations)
DESCRIPTION DESCRIPTION product description product description
END: END: marks the end of a "GROUP:" or "COMMON:"; one "END:" marker is used to jointly end a "GROUP:" and an included "COMMON:" marks the end of a "GROUP:" or "COMMON:"; one "END:" marker is used to jointly end a "GROUP:" and an included "COMMON:"
FILE FILE type of file (possible values: DBCONFIG, UPDCONFIG, CHAIN, VERSION, TABLE) type of file (possible values: DBCONFIG, UPDCONFIG, CHAIN, VERSION, TABLE) D D U U C C V V T T
FLAVOR FLAVOR product instance flavor product instance flavor Note: To easily accommodate flavor-neutral setup functions in a table file, FLAVOR can take the value ANY, but only in a table file. Note: To easily accommodate flavor-neutral setup functions in a table file, FLAVOR can take the value ANY, but only in a table file.
GROUP: GROUP: groups together multiple instances; all entries subsequent to this "GROUP:" are part of it until an "END:" marker is reached groups together multiple instances; all entries subsequent to this "GROUP:" are part of it until an "END:" marker is reached
INFO_SOURCE_DIR INFO_SOURCE_DIR Default: under the ${UPS_UPS_DIR}/ toInfo directory Default: under the ${UPS_UPS_DIR}/ toInfo directory location of Info files included with instance location of Info files included with instance
INFO_TARGET_DIR INFO_TARGET_DIR directory into which Info files are to be copied directory into which Info files are to be copied D D
MAN_SOURCE_DIR MAN_SOURCE_DIR Default: under the ${UPS_UPS_DIR}/ toman/man directory Default: under the ${UPS_UPS_DIR}/ toman/man directory location of unformatted man page files included with instance location of unformatted man page files included with instance
MAN_TARGET_DIR MAN_TARGET_DIR directory into which formatted man pages are to be copied directory into which formatted man pages are to be copied D D
MODIFIED MODIFIED Default: Current date/time Default: Current date/time last time the associated instance was changed last time the associated instance was changed Note: often has multiple values, one for each declaration/modification (e.g., for subsequent chain declarations) Note: often has multiple values, one for each declaration/modification (e.g., for subsequent chain declarations)
MODIFIER MODIFIER Default: Current user Default: Current user userid of user that modified the instance userid of user that modified the instance Note: often has multiple values, one for each declaration/modification (e.g., for subsequent chain declarations) Note: often has multiple values, one for each declaration/modification (e.g., for subsequent chain declarations)
ORIGIN ORIGIN master source file; see option -D in Chapter 25: Generic Command Option Descriptions master source file; see option -D in Chapter 25: Generic Command Option Descriptions
PRODUCT PRODUCT product name product name
PROD_DIR PROD_DIR product root directory (usually defined relative to PROD_DIR_PREFIX, below) product root directory (usually defined relative to PROD_DIR_PREFIX, below)
PROD_DIR_PREFIX PROD_DIR_PREFIX product root directory prefix (area where all or most product instances are maintained) product root directory prefix (area where all or most product instances are maintained) D D
QUALIFIERS QUALIFIERS additional instance specification information often used to indicate compilation options used by developer additional instance specification information often used to indicate compilation options used by developer Notes: appears immediately after a FLAVOR in these files, and is coupled with it to complete the instance identification (see 27.2.3 Qualifiers: Use in Instance Matching ) Notes: appears immediately after a FLAVOR in these files, and is coupled with it to complete the instance identification (see 27.2.3 Qualifiers: Use in Instance Matching )
SETUPS_DIR SETUPS_DIR location of setups.[c]sh files and other UPS initialization files location of setups.[c]sh files and other UPS initialization files D D
STATISTICS STATISTICS flag to record statistics for specified products flag to record statistics for specified products See 28.6.3 STATISTICS for usage information. See 28.6.3 STATISTICS for usage information. D D
TABLE_DIR TABLE_DIR Default: search path (see section 29.4 Determination of ups Directory and Table File Locations ) Default: search path (see section 29.4 Determination of ups Directory and Table File Locations ) location of table file location of table file
TABLE_FILE TABLE_FILE name of table file (relative to TABLE_DIR) name of table file (relative to TABLE_DIR)
UNWIND_ARCHIVE_ FILE UNWIND_ARCHIVE_ FILE (a UPD keyword used only on distribution server configurations) (a UPD keyword used only on distribution server configurations) absolute path to directory in which to unwind archive file (tar file) of product absolute path to directory in which to unwind archive file (tar file) of product
UNWIND_PROD_DIR UNWIND_PROD_DIR (a UPD keyword) absolute path to directory where product gets unwound (a UPD keyword) absolute path to directory where product gets unwound
UNWIND_TABLE_DIR UNWIND_TABLE_DIR (a UPD keyword) absolute path to directory where the table file gets unwound (a UPD keyword) absolute path to directory where the table file gets unwound
UNWIND_UPS_DIR UNWIND_UPS_DIR (a UPD keyword) absolute path to directory where the ups directory gets unwound (a UPD keyword) absolute path to directory where the ups directory gets unwound
UPD_USERCODE_DB UPD_USERCODE_DB Database containing UPD_USERCODE_DIR (set internally) Database containing UPD_USERCODE_DIR (set internally)
UPD_USERCODE_DIR UPD_USERCODE_DIR Directory where UPD configuration files are maintained Directory where UPD configuration files are maintained D D
UPS_ARCHIVE_FILE UPS_ARCHIVE_FILE (a UPD keyword used only on distribution server configurations) (a UPD keyword used only on distribution server configurations) archive file (tar file) location that UPD specifies in archive file (tar file) location that UPD specifies in ups declare -T ftp://host${UPS_ARCHIVE_FILE} ups declare -T ftp://host${UPS_ARCHIVE_FILE}
UPS_DB_VERSION UPS_DB_VERSION UPS database version UPS database version D D
UPS_DIR UPS_DIR Default: ${UPS_PROD_DIR}/ups if directory exists there Default: ${UPS_PROD_DIR}/ups if directory exists there location of ups directory (if not absolute path, then taken relative to PROD_DIR, if specified) location of ups directory (if not absolute path, then taken relative to PROD_DIR, if specified)
UPS_PROD_DIR UPS_PROD_DIR (a UPD keyword) product root directory that UPD specifies in the ups declare -r option; should be defined relative to PROD_DIR_PREFIX for portability (a UPD keyword) product root directory that UPD specifies in the ups declare -r option; should be defined relative to PROD_DIR_PREFIX for portability
UPS_TABLE_DIR UPS_TABLE_DIR (a UPD keyword) table file directory that UPD specifies in the ups declare -M option (a UPD keyword) table file directory that UPD specifies in the ups declare -M option % %{font-weight: bold;color: #000000}Normally this should not be set ! Since UPS_TABLE_DIR must be an absolute path, the declaration becomes non-portable if you set this location. % %{font-weight: bold;color: #000000}Normally this should not be set ! Since UPS_TABLE_DIR must be an absolute path, the declaration becomes non-portable if you set this location.
UPS_THIS_DB UPS_THIS_DB (a UPD keyword) the database into which UPS declares the product (i.e., the directory that UPD specifies in the ups declare -z option). (a UPD keyword) the database into which UPS declares the product (i.e., the directory that UPD specifies in the ups declare -z option).
UPS_UPS_DIR UPS_UPS_DIR (a UPD keyword) ups directory that UPD specifies in the ups declare -U option, taken relative to ${UNWIND_PROD_DIR} unless an absolute path is given; usually defined as ups. (a UPD keyword) ups directory that UPD specifies in the ups declare -U option, taken relative to ${UNWIND_PROD_DIR} unless an absolute path is given; usually defined as ups.
UPS_TABLE_FILE UPS_TABLE_FILE (a UPD keyword) table file name that UPD specifies in the ups declare -m option (a UPD keyword) table file name that UPD specifies in the ups declare -m option
USER USER current username current username
VERSION VERSION product version product version
_UPD_OVERLAY _UPD_OVERLAY main product name for overlaid product main product name for overlaid product Note: This keyword is user-defined from UPS 's point of view. It is included here because it is configured and used by UPD . Its use with overlaid products is described in section 28.6.6 _UPD_OVERLAY . Note: This keyword is user-defined from UPS 's point of view. It is included here because it is configured and used by UPD . Its use with overlaid products is described in section 28.6.6 _UPD_OVERLAY .
|
|
|
T T

  28.5 Syntax for Assigning Keyword Values

  • Any keyword value that has multiple values uses a colon ( : ) to separate the subvalues. The value (i.e., the list of subvalues) may be surrounded by double quotation marks ( "..." ). Blanks within the double-quoted value are ignored; they are neither required nor prohibited.
    p<>.   For example, the following are all equivalent:

  QUALIFIERS = debug:optimize

  QUALIFIERS = "debug:optimize"

  QUALIFIERS = " debug: optimize"

  • Whitespace is ignored except within the keyword values for DESCRIPTION, DECLARER and MODIFIER
  • Leading whitespace is ignored.
  • There are no line continuation characters; the entire keyword definition or function must appear on a single line.
  • The "at" character (@) is defined for use with the keywords COMPILE_FILE, PROD_DIR, UPS_DIR and TABLE_FILE. See section 28.6 Usage Notes on Particular Keywords .

  28.6 Usage Notes on Particular Keywords

  28.6.1 COMPILE_DIR, COMPILE_FILE and @COMPILE_FILE

  COMPILE_DIR

  the directory in which the compile file resides (see Chapter 38: Use of Compile Scripts in Table Files )

  COMPILE_FILE

  the name of the file containing precompiled functions

  @COMPILE_FILE

  the entire path to the file containing precompiled functions

  28.6.2 PROD_DIR_PREFIX, PROD_DIR and @PROD_DIR

  PROD_DIR_PREFIX

  is generally set to the root of the path shared by all the products.

  PROD_DIR

  is the path that gets specified when the particular product instance is declared; it is usually (but not always) a relative path that gets tacked onto PROD_DIR_PREFIX.

  @PROD_DIR

  is a shorthand to request the entire path for the directory where the product is installed (usually equivalent to PROD_DIR_PREFIX/PROD_DIR).

  If PROD_DIR_PREFIX is not defined on your system, then PROD_DIR should represent the entire path, in which case PROD_DIR and @PROD_DIR are identical.

  Compare these commands and their output:

% ups list -K PROD_DIR_PREFIX sam_docs
"/afs/fnal.gov/ups/prd" 
% ups list -K PROD_DIR sam_docs
"sam_docs/v1_0/NULL" 
% ups list -K @PROD_DIR sam_docs
"/afs/fnal.gov/ups/prd/sam_docs/v1_0/NULL" 

  28.6.3 STATISTICS

  The STATISTICS keyword is provided to allow recording of the following statistics on product usage and UPS database access:

  • Userid of person executing UPS/UPD command
  • Date and time
  • Which command was executed (including options and arguments)
  • Which product instance was selected by command

  This keyword can appear in a product's version file and/or in the UPS database configuration file, thus providing a great deal of flexibility in choosing which products/instances to monitor.

  Use in a Version File

  When the STATISTICS keyword is present in a version file, it must be included with each specific instance which is to be monitored. If the STATISTICS keyword is located before any FLAVOR and/or QUALIFIERS keywords (these keywords separate out different instances), then it is ignored. In a version file, this keyword should have no value assigned.

  Use in a Database Configuration File

  When the STATISTICS keyword appears in the database configuration file, it needs a value. (If it has no value, it is ignored.) Its value is a colon-separated list of the products (name only) on which to record statistics (e.g., STATISTICS = "tcl:tk:cern" ). The value * (asterisk) indicates that statistics are to be gathered on all products in the database.

  Statistics Output

  For a given product being monitored, statistics data for the product get recorded in a file whose name is the same as the product. If the product has dependencies, data also get recorded for them in their own product-specific files, and the data include the parent product name and version number. The data get recorded only when the UPS/UPD command in question has succeeded (i.e., when the temporary file has been created, but not yet sourced).

  The statistics output files for all the monitored products and their dependencies reside in a special directory associated with the UPS database, namely $PRODUCTS/.upsfiles/statistics . This makes it easy to determine which products are being monitored, and only one directory needs to be made world-writable.

  As an example of the statistics data that get recorded, let's look at the tcl product. It is a dependency of tk . Data that are recorded when an instance of tcl is accessed independently look like this:

"tcl" "v8_0" "Linux" "" "" "user1" "2013-03-18 15.22.36 GMT" "setup" 

  Data that are recorded for tcl when an instance of tk is accessed look like this:

"tcl" "v8_0" "Linux" "" "" "user1" "2013-03-18 15.22.36 GMT" "setupRequired tk v8_0" 

  Statistics Example

  The system can be set up to log a subset of information, such as the version of the software and the top level product, instead of everything. Once the data has been collected, users can grab parts of the output file for specific information as shown in the following example:

set "STATISTICS = larsoft:lbne_software" and run commands.

Then

% cd /path/to/db/.upsfiles/statistics

% cat * | cut -d '"' -f 12  | sort | uniq -c
to get a count of how many times each user set things up

The above command tells the 'cut' command to use a double quote as a separator. Ups lists things in the log file like this: "prod" "vers" "flavor" "quals" "" "user" "date" "command"
p<>. The 'cut' command, when it uses a double quote as a separator, counts the whitespace (and the empty string preceding the first double quote) as fields, too. So to get the username, which is the 6th item in the file, use:

cut -d '"' -f 12
To get the hour of the day, we use one cut to extract the date, 
   cut -d '"' -f 14
     and then use another cut command to pull the hour out by column number
   cut -c 12-13

    We then run that text into
     sort | uniq -c
    to give a count of how many times each item occurs.

So, to put it all together, use:
% cat * | cut -d '"' -f 14  | cut -c 12-13 | sort | uniq -c
to get count by hour of the day

  28.6.4 TABLE_FILE and @TABLE_FILE

  TABLE_FILE represents only the name of the table file, not its path. @TABLE_FILE is the entire path for the table file. Compare these commands and their output:

% ups list -Ktable_file sam_docs
"v1_0.table" 
% ups list -K@table_file sam_docs
"/afs/fnal.gov/ups/db/sam_docs/v1_0.table" 

  See section 29.4 Determination of ups Directory and Table File Locations for information on how UPS determines the table file directory.

  28.6.5 UPS_DIR and @UPS_DIR

  UPS_DIR represents the location of the product's ups directory. If it is not an absolute path, then it is taken relative to @PROD_DIR (as shown in the example below). @UPS_DIR is the absolute path. Compare these commands and their output:

% ups list -K @PROD_DIR sam_docs
"/afs/fnal.gov/ups/prd/sam_docs/v1_0/NULL" 
% ups list -Kups_dir sam_docs
"ups" 
% ups list -K@ups_dir sam_docs
"/afs/fnal.gov/ups/prd/sam_docs/v1_0/NULL/ups" 

  28.6.6 _UPD_OVERLAY

  The _UPD_OVERLAY keyword defined in UPD is provided for inclusion in the table file of each overlaid product. Overlaid products are introduced in section 2.3.7 Product Overlays and discussed again for developers in section 17.2.4 Overlaid Products . _UPD_OVERLAY takes as its value the main product name in double quotes. Its presence indicates that the product is an overlaid product maintained in the root directory of the main product listed as the keyword's value. For example, the table files for the products cern_bin , cern_ups , and cern_lib would contain the following keyword line:

  _UPD_OVERLAY = "cern"

  UPD would then use cern as the product name when determining the root directory.

 

TOC PREV NEXT

Last revised in July 2014