Project

General

Profile

Bug #3123

CatalogInterface (do)OutputFileClosed, (do)OutputModuleInitiated are not called

Added by Marc Mengel over 7 years ago. Updated over 7 years ago.

Status:
Closed
Priority:
Normal
Category:
I/O
Target version:
Start date:
11/13/2012
Due date:
% Done:

100%

Estimated time:
Occurs In:
Scope:
Internal
Experiment:
-
SSI Package:
Duration:

Description

The current art code (1.02.05) calls the CatalogInterface when an output file is opened
via (do)OutputFileOpened, but does not call (do)OutputFileCLosed when the file is closed.
The latter is the call that we actually need for data hanlding code, as this lets us know
both what the file written is actually called, and that it is ready for transfer. At least,
it does so when working with the Armistice Day release of Nova. I'm guessing this call should
go in OutputModule::reallyCloseFile()(?)

There is also a (do)OutputModuleInitiated method which is not called. It appears to be there
to let us see the parameter set on the output module, so we can maybe decide if we're interested
in it, and it would appear we should call it someplace like the OutputModule(fhicl::ParameterSet const& pset) constructor (i.e. a user could add samIgnore: true or some such on an output file).
The other possibility is that this is getting called upstream of the OutputModule constructor, but due to the order of initialization of modules, the call is going to the TrivialCatalogInterface(?)

At any rate, getting the information these calls provide will let us prototype adding the return of
output files to the SAM "ifdh_art" code.

I'm assigning this to Chris directly, since he asked me to make this ticket in email.


Related issues

Related to art - Feature #3097: IFCatalogInterface support in NOvARawInputSourceClosed11/01/201211/09/2012

History

#1 Updated by Christopher Green over 7 years ago

  • Description updated (diff)
  • Target version set to 1.02.06

As I said in the initial email: outputModuleInitiated() was an omission: this can be added easily.

The call to outputFileClosed() was placed where it should have been called, but there appears to be a difference between how I thought this part of the event management / state machine functioned, and what actually happened. This will need some analysis. I will do my best to have this fixed by the end of the week, but I cannot promise.

#2 Updated by Marc Mengel over 7 years ago

I think with the outputModuleInitiated() call, we can get the filename from the config dictionary, and note the filename to be copied back at the end of the job, so I think we
can get base functionality with just that.

If/when we want to support asynchronously copying output files back the moment they're closed, the outputFileClosed() call will be needed; but that isn't needed right away. So I guess I'm saying don't hold up a release on the outputFileClosed() fix if we can get the outputModuleInitiated() bit in there.

#3 Updated by Marc Mengel over 7 years ago

I must make a correction; in 1.02.05, the doOutputFileClosed call is (if the program terminates without errors) being made, but it is getting passed the name of the last input file, not the output file.

#4 Updated by Christopher Green over 7 years ago

  • % Done changed from 0 to 90

I have updated ~greenc/work/cet-is/products/art/v1_02_06 with what I believe is a fix for the problem of wrong file name to doOutputFileClosed(): please verify.

In addition:

  • Call to outputModuleInitiated() was added with e0436f1
  • Call to eventSelected() was added with ac8f840. Please note that if the size() of the HLTGlobalStatus trigger results object was 0, that means that there was no trigger results object (likely because there were no filter modules and/or no trigger path). This is a valid and quite common condition.

#5 Updated by Marc Mengel over 7 years ago

Yes! That works well. I'm now seeing:

IFCatalogInterface doOutputModuleInitiated, got keys:
dataTier,
fastCloning,
fileName,
module_label,
module_type,
streamName
...
IFCatalogInteface doOutputFileOpened: out1
...
IFCatalogInteface doOutputFileClosed: out1, cosmic_reco.root

So that looks excellent.

#6 Updated by Christopher Green over 7 years ago

  • % Done changed from 90 to 100

All of the required fixes are on the develop branch, and also on the v1_02_05-branch branch, so I believe this issue is resolved and v1_02_06 is ready to be cut. Please reply immediately if this is not the case.

#7 Updated by Christopher Green over 7 years ago

  • Status changed from New to Resolved

#8 Updated by Christopher Green over 7 years ago

  • Status changed from Resolved to Closed


Also available in: Atom PDF