Feature #4031

For DS-50 DAQ/artdaq, support customizable output files

Added by Kurt Biery over 7 years ago. Updated almost 7 years ago.

Target version:
Start date:
Due date:
% Done:


Estimated time:
(Total: 4.00 h)
Spent time:
(Total: 23.25 h)
SSI Package:
Duration: 15


There have been requests for features like the following:
  1. name the disk file based on a pattern specified in the configuration file, and allow the pattern to include fields such as the current run number, the current subrun number, and a timestamp.
  2. allow users to specify a maximum file size or a maximum number of events per file or a maximum amount of time for a file to be open. The idea here would be that after the configurable size or number of events or amount of time, the current file would be closed and a new one opened, and new events would be written to the new file.
  3. after a file is successfully closed, move it to a different (configurable) directory

In all of these cases, I think that it would be useful to discuss the needs and the available options a bit more. For example, in the case of the second request, it may be more robust to have the System Control application send Pause & Resume commands at the appropriate times to get the files to have the correct size, number of events, or time range.

For the first request, it would be good to discuss the patterns that would need to be supported and how they would be specified.

Here is a note from Jim on the first request:
  • This is experiment-specific configuration. A proposal that is more detailed is needed here - one that includes an example art fhicl configuration fragment that would specify the name. An alternative to changing fundamental behavior for naming conventions would be to use an experiment-specific service that renames the files to an appropriate name when a "file closed" event is generated within art.
And, here is a note from Jim on the third request:
  • An experiment-specific art service may be able to easily be used for this using the existing APIs.


Feature #4357: Implement on-close file configurable file renaming [ds50daq-related]ClosedChristopher Green

Feature #4625: Implement configurable temporary directory for in-progress output file.ClosedChristopher Green

Feature #4360: Add a new postCloseOutputFile signal [ds50daq-related]ClosedChristopher Green


#1 Updated by Kurt Biery over 7 years ago

  • Due date set to 07/31/2013
  • Priority changed from Normal to Urgent

Notes from Alessandro/Jim/Kurt discussion:

It is really bad when, after a full night of running, the system crashes and the very large data file is lost.

Event numbers must not reset to one when a subrun changes.

Pausing and resuming has undesirable consequences for the livetime.

Various levels of triggering a new subrun: pause/resume from SystemControl, internal pause/resume, simply having the aggregator create the subrun messages.

Alessandro asks: why are subruns needed to get a new file? Jim - it is natural to do it there given the bookkeeping that art needs to do.

Let's say that a subrun is simply a checkpoint so that it is easy for Alessandro to describe to his colleagues.

Alessandro says that we can keep the new subrun when pause/resume happens - SystemControl will not make pause/resume very visible.

Jim suggests a watchdog that watches various quantities and emits a new subrun signal.

This will be a general feature of artdaq.

On the subject of file naming: Alessandro would like to be able to specify a pattern (with %r for run, %s for subrun, etc.)
He also wants the ability to move a file once it is closed.

There is some short investigation that needs to be done with this.

Alessandro would like to meet in two weeks to checks progress.

#2 Updated by Christopher Green over 7 years ago

  • Category set to I/O
  • Status changed from New to Resolved
  • Assignee set to Christopher Green
  • Target version set to 1.08.00
  • Scope set to Internal
  • Experiment DarkSide added
  • SSI Package art added

We believe this issue to be resolved due to the work done toward the subtasks.

#3 Updated by Lynn Garren almost 7 years ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF