Project

General

Profile

Feature #9830

Inherit sam metadata

Added by Herbert Greenlee over 5 years ago. Updated over 4 years ago.

Status:
Closed
Priority:
High
Assignee:
Category:
Metadata
Target version:
Start date:
08/17/2015
Due date:
% Done:

100%

Estimated time:
8.00 h
Spent time:
Scope:
Internal
Experiment:
MicroBooNE
SSI Package:
art
Duration:

Description

Art service FileCatalogMetadata sets internal sam metadata parmaeters run_type and file_type, which are currently set by fcl parameters. We request a feature to teach FileCatalogMetadata to inherit the run_type and file_type from the input file.

History

#1 Updated by Christopher Green over 5 years ago

  • Status changed from New to Feedback
  • SSI Package art added
  • SSI Package deleted ()
Some questions:
  1. If the command line / configuration and the input file disagree, what should take precedence?
  2. If, among the specified input files, there is more than one value for run_type or file_type, what should happen?

#2 Updated by Christopher Backhouse over 5 years ago

NOvA is interested in this too.

My answers would be
1. Command-line/config should take precedence
2. Conflict between input files should be an error (unless command-line/config specifies an override).

But obviously Herb should give his preferences too.

We tried to attack this from our side by putting logic in our code to copy these fields over, but there's some reason why these are treated specially by art and that can't be done. I don't recall the details.

#3 Updated by Herbert Greenlee over 5 years ago

There should be a fcl parameter that will specify whether to get run_type and file_type from the input data, or to use a fcl override. I'd say specifying either paramater on the command line should override input data.

Here's a use case we will probably use: Read random triggered data as input, and merge (mix) mc data. Resulting file should have file_type=mc.

With respect to reading multiple input files with one output file, then conflicting input sam metadata should trigger an error.

#4 Updated by Kyle Knoepfel almost 5 years ago

  • Status changed from Feedback to Accepted
  • Assignee set to Paul Russo

We intend to include this in the next art 1.17 release.

#5 Updated by Kyle Knoepfel almost 5 years ago

  • Assignee changed from Paul Russo to Kyle Knoepfel

#6 Updated by Kyle Knoepfel almost 5 years ago

  • Status changed from Accepted to Assigned
  • Estimated time set to 8.00 h

#7 Updated by Kyle Knoepfel almost 5 years ago

  • Status changed from Assigned to Resolved
  • % Done changed from 0 to 100

This feature has been implemented with commits art:5e0736cf and art:a4d24f26. The use cases are described below.

Command-line

The file_type and run_type metadata can be inherited from an input file in two ways:

art -c config.fcl -s input.root -o output.root --sam-inherit-file-type --sam-inherit-run-type
art -c config.fcl -s input.root -o output.root --sam-inherit-metadata   # equivalent to above

If it is desired that only one of the metadata fields is inherited, this can be done via:

art -c config.fcl -s input.root -o output.root --sam-inherit-file-type --sam-run-type="MCChallenge" 
art -c config.fcl -s input.root -o output.root --sam-inherit-run-type --sam-file-type="MC" 

Any of the following command-line invocations are errors:

art -c config.fcl -s input.root -o output.root --sam-inherit-file-type --sam-file-type="MC"         # Error - conflicting options
art -c config.fcl -s input.root -o output.root --sam-inherit-run-type  --sam-run-type="MCChallenge" # Error - conflicting options
art -c config.fcl -s input.root -o output.root --sam-inherit-metadata  --sam-file-type="MC"         # Error - conflicting options
art -c config.fcl -s input.root -o output.root --sam-inherit-metadata  --sam-run-type="MCChallenge" # Error - conflicting options

Note that additional program options may need to be specified depending on which combinations of SAM program options are used.

FHiCL file

One can specify in a configuration file that SAM metadata should be inherited from the input file. The following are valid configurations (where only the parameters relevant to this issue are being specified):

# 1. Inherit both "fileType" and "runType" 
services.FileCatalogMetadata: {
   metadataFromInput: ["fileType", "runType"]
}
# 2. Inherit only "fileType" 
services.FileCatalogMetadata: {
   metadataFromInput: ["fileType"]
   runType: "MCChallenge" 
}
# 3. Inherit only "runType" 
services.FileCatalogMetadata: {
   metadataFromInput: ["runType"]
   fileType: "MC" 
}
# 4. Inherit nothing
services.FileCatalogMetadata: {
   metadataFromInput: []
   fileType: "MC" 
   runType: "MCChallenge" 
}

Just as in the command-line options, it is an error to specify conflicting FHiCL parameters. For example, the following results in an ambiguity that triggers a configuration error:

# Error example: Cannot specify 'fileType' when it is being inherited 
#                as specified in the 'metadataFromInput' line
services.FileCatalogMetadata: {
   metadataFromInput: ["fileType", "runType"]
   fileType: "MC" # Error
}

For the full set of specifiable FHiCL parameters, type art --print-description FileCatalogMetadata.

Command-line vs. FHiCL configuration

Specified command-line options take precedence over FHiCL parameters. It is therefore not an error to have a command-line option that conflicts with the configuration--in such a case, the command-line invocation replaces the FHiCL specification.

Conflicting input metadata

If a particular SAM metadata field is inherited by input, and the value of that field for the second input file differs with respect to that of the first input file, an exception is thrown.

#8 Updated by Kyle Knoepfel almost 5 years ago

  • Target version set to 1.17.06

#9 Updated by Kyle Knoepfel over 4 years ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF