For DS-50 DAQ/artdaq, support customizable output files
- name the disk file based on a pattern specified in the configuration file, and allow the pattern to include fields such as the current run number, the current subrun number, and a timestamp.
- allow users to specify a maximum file size or a maximum number of events per file or a maximum amount of time for a file to be open. The idea here would be that after the configurable size or number of events or amount of time, the current file would be closed and a new one opened, and new events would be written to the new file.
- after a file is successfully closed, move it to a different (configurable) directory
In all of these cases, I think that it would be useful to discuss the needs and the available options a bit more. For example, in the case of the second request, it may be more robust to have the System Control application send Pause & Resume commands at the appropriate times to get the files to have the correct size, number of events, or time range.
For the first request, it would be good to discuss the patterns that would need to be supported and how they would be specified.
- This is experiment-specific configuration. A proposal that is more detailed is needed here - one that includes an example art fhicl configuration fragment that would specify the name. An alternative to changing fundamental behavior for naming conventions would be to use an experiment-specific service that renames the files to an appropriate name when a "file closed" event is generated within art.
- An experiment-specific art service may be able to easily be used for this using the existing APIs.
#1 Updated by Kurt Biery over 6 years ago
- Due date set to 07/31/2013
- Priority changed from Normal to Urgent
Notes from Alessandro/Jim/Kurt discussion:
It is really bad when, after a full night of running, the system crashes and the very large data file is lost.
Event numbers must not reset to one when a subrun changes.
Pausing and resuming has undesirable consequences for the livetime.
Various levels of triggering a new subrun: pause/resume from SystemControl, internal pause/resume, simply having the aggregator create the subrun messages.
Alessandro asks: why are subruns needed to get a new file? Jim - it is natural to do it there given the bookkeeping that art needs to do.
Let's say that a subrun is simply a checkpoint so that it is easy for Alessandro to describe to his colleagues.
Alessandro says that we can keep the new subrun when pause/resume happens - SystemControl will not make pause/resume very visible.
Jim suggests a watchdog that watches various quantities and emits a new subrun signal.
This will be a general feature of artdaq.
On the subject of file naming: Alessandro would like to be able to specify a pattern (with %r for run, %s for subrun, etc.)
He also wants the ability to move a file once it is closed.
There is some short investigation that needs to be done with this.
Alessandro would like to meet in two weeks to checks progress.
#2 Updated by Christopher Green over 6 years ago
- Category set to I/O
- Status changed from New to Resolved
- Assignee set to Christopher Green
- Target version set to 1.08.00
- Scope set to Internal
- Experiment DarkSide added
- SSI Package art added
We believe this issue to be resolved due to the work done toward the subtasks.