Feature #8593: Improve flexibility of output file handling
Configurable output file switching for artdaq
(Topics similar to what is described here have been discussed in the past, so I apologize if this new Issue duplicates an existing one.)
The feature that we would like to have is the ability for an output module that is writing data to disk in artdaq to close one file and open a new one based on thresholds that are configured in the output module. The types of thresholds that have been discussed in the past are file size, file age, and the number of events in the file. (As a reminder, the files that are currently written by artdaq are in ROOT format.)
At the moment, this behavior is obtained by sending suitable endSubrun messages to the art thread inside the Aggregator, but it would be nice for the output module to handle the automatic file closing without needing specialized behavior from the DAQ system.
The use case that highlights this issue anew is the desire to write multiple streams from the Aggregator in the LBNE 35t DAQ. We know that writing multiple output streams will be easily supported by adding RootOutputModules with suitable configuration to the Aggregator. However, given the file-closing mechanism that we are using in artdaq now, we believe that the closing and opening of the files in the different streams will be tied together, and this may have undesirable side-effects. For example, if one stream is accepting significantly more events than another, and a file-size threshold is in effect, the file-closing needs of the larger stream will result in many small files for the smaller stream.
In the past, our discussions of topics like this have included concerns that an arbitrary closing of one file and opening of a new one may result in events not being perfectly ordered between files. For example in a filtered stream, file1 could have events 1, 3, 5, 12, and 14, and file2 could have events 7, 9, 17, and 20. This situation could be surprising to users who might expect file1 to have events 1, 3, 5, 7, and 9, and file2 to to have events 12, 14, 17, and 20. However, my sense is that users are becoming accustomed to reading in all files in a file set when they run their nearline and offline jobs, so I'm hopeful that this issue is not a big one. (Of course, we should check with users.)
As part of this request, we are not requesting any change to the current behavior of an endSubrun causing the closing of the current file in all streams and the creation of a new file per stream if/when a new subrun begins. At the moment, subruns are tied to Pause&Resume transitions in artdaq. We may choose to break that connection at some point, but we will always want the ability to end a subrun and start a new one (and have files get closed). Of course, in that case, we will provide some other mechanism in the DAQ system to send the endSubrun message to the art thread in the Aggregator.
Thanks for considering this request,
#1 Updated by Kyle Knoepfel almost 5 years ago
- Status changed from New to Accepted
We note that this issue is connected with others, and we will have to evaluate how much change is necessary to do this to come up with a reasonable time estimate. We will update this issue with the list of associated issues.
#12 Updated by Kyle Knoepfel over 3 years ago
- Status changed from Assigned to Resolved
- % Done changed from 0 to 100
- SSI Package art added
- SSI Package deleted (
The ability to conditionally rollover to a new output file, independently of other output modules, has been implemented for the art:version:"Arcturus" release. The supported conditions include switching on:
- Maximum file-size,
- Maximum number of events written to the output module, and
- the ability to force a file switch on a specified boundary:
Note that the ability to switch to a new output file based on the file "age" will not be implemented for art:version:"Arcturus" -- a new issue (#12566) has been opened for that feature.
Please see the documentation for the next feature release of art. Placeholder documentation can be found here.
Implemented with many commits. First merge commit is art:97485ee2.