cet-is » art

We believe we can provide an implementation that places reasonable demands on the user without introducing unreasonable entanglements within art.

This feature has been largely committed with commit art:5c502b4c. Some remaining issues may need to be discussed with the stakeholders and with the art team.

Documentation regarding how it can be enabled is forthcoming.

Please see the attached files that include versions of TFileService and TFileDirectory, which support the file-switching behavior you seek. In addition, there is a CMakeLists.txt file that gives you the appropriate library dependencies, assuming you're using mrb/cetbuildtools.

There will likely be some adjustments required whenever the updated TFileService is released with art 2.10.00. For that reason, I encourage you to rename your copy of the service to something else. This will avoid unintentional collisions with the officially-released version.

We spent considerable time trying to find a way to allow TFileService to seamlessly switch to a new input file without users needing to change their C++ source code. Unfortunately, since TFileService exposes various cached objects and allows users to cache pointers to ROOT objects, it was not feasible to do this given ROOT's memory management. Even though it may be possible to retain the pointer values for anything associated with a TFileService file, whenever a new file is opened, the directory pointer cached by the TObjects would be incorrect. Adjusting that pointer is difficult as is recreating a new TFile object at the same memory address as the old file without causing memory corruption issues.

For that reason, we have added new interface to TFileService that allows users to specify which actions should be performed after a file switch has occurred. This gives users the opportunity to update their pointers at the right time, but it still conceals the file-switching mechanics that are necessary for the framework to handle. See below for documentation.

Job configuration¶

To configure the TFileService to switch to a new output file for a given condition, the optional fileProperties table can be specified, just as it can be specified for any RootOutput module:

## Any parameters prefaced with '#' are optional.

TFileService: {

   closeFileFast: true  # default

   fileName: <string>

   tmpDir: "<parent-path-of-filename>"  # default

 # fileProperties: {
 #
 #    maxEvents: 4294967295  # default
 #
 #    maxSubRuns: 4294967295  # default
 #
 #    maxRuns: 4294967295  # default
 #
 #    maxInputFiles: 4294967295  # default
 #
 #    ## Maximum size of file (in KiB)
 #
 #    maxSize: 2130706432  # default
 #
 #    ## Maximum age of output file (in seconds)
 #
 #    maxAge: 4294967295  # default
 #
 #    ## The 'granularity' parameter specifies the level at which
 #    ## an output file may be closed, and thereby the granularity
 #    ## of the file.  The following values are possible:
 #    ##
 #    ##     Value        Meaning
 #    ##    =======================================================
 #    ##    "Event"       Allow file switch at next Event
 #    ##    "SubRun"      Allow file switch at next SubRun
 #    ##    "Run"         Allow file switch at next Run
 #    ##    "InputFile"   Allow file switch at next InputFile
 #    ##    "Job"         File closes at the end of Job
 #    ##
 #    ## For example, if a granularity of "SubRun" is specified, but the
 #    ## output-module has reached the maximum events written to disk (as
 #    ## specified by the 'maxEvents' parameter), the output module will NOT
 #    ## switch to a new file until a new SubRun has been reached (or
 #    ## there are no more Events/SubRuns/Runs to process).
 #
 #    granularity: "Event"  # default
 # }
}

The comment needs to be updated so that it does not refer to the "output module", but to a "service" instead.

N.B. If the fileProperties parameter is specified, then the only modules using TFileService that can be used in the job are those that implement the changes described in the next section.

Usage in modules¶

If you want your module to be usable whenever a user enables TFileService file-switching, then it must provide a file-switch callback. For example:

TestTFileService::TestTFileService(Parameters const& p)
  : EDAnalyzer{p}
{
  ServiceHandle<TFileService> fs;
  fs->registerFileSwitchCallback(this, &TestTFileService::setRootObjects);
  setRootObjects();
}

void
TestTFileService::setRootObjects()
{
  ServiceHandle<TFileService> fs;
  h1_ = fs->make<TH1F>("test1", "test histogram #1", 100, 0., 100.);
}

In this example, we have provided the 'void setRootObjects()' function as a callback to the TFileService. After a file-switch has occurred, the callback will be invoked, resetting h1_ to a new histogram.

N.B. The callbacks are invoked only after a file switch has occurred--they are not invoked after the constructor of the module has been called. Hence why we have called setRootObjects in the constructor as well as provided it as a callback to the TFileService.

If the job is configured to switch TFileService files, and one of the modules does not register a callback, you will see an error like (e.g.):

---- Configuration BEGIN
  A TFileService error occured while attempting to make a directory or ROOT object.
  File-switching has been enabled for TFileService.  All modules must register
  a callback function to be invoked whenever a file switch occurs.  The callback
  must ensure that any pointers to ROOT objects have been updated.

    No callback has been registered for module 'a1'.

  Contact artists@fnal.gov for guidance.
---- Configuration END

An unfortunate consequence¶

Because art does not directly interact with any objects created via the TFileService, art cannot be smart about when to open a ROOT file--art must be cautious. To that end, a ROOT file is created (a) whenever the TFileService is constructed, and (b) immediately after a file has been closed during a file switch. This means that in some cases, it is possible to get an extra file that is not necessarily desired. For example:

• TFileService is configured to switch after 10 events
• The job is configured to process only 10 events
• After the 10th event is processed, TFileService closes one file and opens a new one
• The endSubRun and endRun calls are made to all the modules
• The newly opened file is then closed, even though the histograms may be empty if none of them were filled during endSubRun or endRun.

This consequence is largely unavoidable. It may be possible to improve the situation, but there is no apparent way to do so right now without imposing additional restrictions on TFileService usage.

Future directions¶

There was a request for the art team to be cognizant of user-defined conditions that would trigger a file-switch. Although we have not exposed any interface to allow that at this time, it would be straightforward to add the feature in a non-breaking way in the future.

The directory-setting when invoking callbacks was not being done properly. This has been fixed--please see the attached patch file.