A convenience script could be provided in DAQInterface to tell users where their root files are
DAQInterface currently has a couple of convenience scripts, documented in its manual, which help users look at the output of their run (e.g., "show_logfile_for_process.sh", which will point users to an artdaq process's logfile given the run number and process label). It might be nice to have a script which lists the output root files from a run, so users won't need to scrounge around in the saved eventbuilder or datalogger FHiCL documents to figure out what directory the root files were written to (and then have to look in the boot file to figure out which hosts the eventbuilders and dataloggers were on!). E.g., in the case of my personal run 2898 on the mu2edaq cluster
could return something like
Getting ambitious, the script could even do things like tell you if it expected a root file but didn't find one (e.g., if there were no events in the run, or a datalogger timed out or crashed), as well as tell you how many events were in the root file. Perhaps it could even call rawEventDump? Suggestions welcome.
JCF: Issue #23262: allow more than one module to write root files in a process as long as only one is used
JCF: Issue #23262: be more robust as to the formatting the run number in the root filename can take
JCF: Issue #23262: have show_rootfiles_for_run.sh fall back on a generic rawEventDump if no-other's available
JCF: Issue #23262: needed to add rawEventDump.fcl for the previous commit for this issue to work
JCF: Issue #23262: make sure show_rootfiles_for_run doesn't expect root files from unused modules
#1 Updated by John Freeman 8 months ago
- % Done changed from 0 to 100
- Status changed from New to Resolved
At the head of feature/23262_root_file_info (c8d28363925d8e8def1fcd65a3f143831f1ba802) the show_rootfiles_for_run.sh script is ready for review. The simplest usage is to just pass it the run number:
show_rootfiles_for_run.sh <run number>
where it will then list the *.root file(s) produced during the run.
For a little more sophistication, you can enter a second, optional argument, which is the # of events you want rawEventDump to print out from each root file:
show_rootfiles_for_run.sh <run number> <# of events to list>
To experience the full capabilities of show_rootfiles_for_run.sh, it's a good idea to test it on a run which had multiple diskwriting artdaq processes spread across different nodes. Please note that if "rawEventDump" isn't already available in the shell AND there's no rawEventDump alias in the DAQ setup script for the run in question, show_rootfiles_for_run.sh will be unable to list the events, even though of course it can still list the root files.
#2 Updated by John Freeman 8 months ago
Gennadiy played around with the script and had some useful insights into how it could be improved:
- Right now, show_rootfiles_for_run.sh, when asked to display one or more events per file, relies on rawEventDump either already being available in the environment or being something that gets alias-ed in the DAQ setup script for the run in question. Would it be possible for it to have a bare-bones, experiment-independent default to fall back on if rawEventDump weren't available in either of those two aforementioned locations?
- show_rootfiles_for_run.sh, when figuring out where to find root files, should have a broader knowledge of the formatting that can be used for filenames in the output modules. E.g., it can understand myfile_%06r.root, but not myfile_%R.root.
- show_rootfiles_for_run.sh currently assumes that if a string of the form "filename: <file that ends in root>" appears in a process's FHiCL document, that it implies that root file gets written. However, this isn't a guarantee - e.g, you could have a RootOutput module which contains this line, but then the module doesn't appear in the my_output_modules path
#4 Updated by Gennadiy Lukhanin 8 months ago
Testing this feature branch on icarus-vst01 as icarus.
The artdaq_daqinterface package was built using buildtool and installed into the /software/products_dev directory on icarus-vst01.
[icarus@icarus-vst01 DAQInterface]$ ups active |grep artdaq_daqinterface artdaq_daqinterface v3_06_01_FB23262 -f NULL -z /software/products_dev
- Running the "show_recent_runs.sh 5" command.
[icarus@icarus-vst01 DAQInterface]$ show_recent_runs.sh 5 Run 240 (Sep 27 14:10) : standard : pmtx01 Run 241 (Sep 27 14:18) : standard : pmtx01 Run 242 (Sep 27 14:22) : standard : pmtx01 Run 243 (Sep 27 14:37) : standard : pmtx01 Run 244 (Sep 27 14:44) : standard : pmtx01
- Testing the "show_rootfiles_for_run.sh 244" script.
[icarus@icarus-vst01 DAQInterface]$ show_rootfiles_for_run.sh 244 Run number is 244 Will show the first 0 events from the root file(s) (if any) produced in the run =======================================icarus-vst02======================================= -rw-r--r-- 1 icarus P-1052 64688189 Sep 27 14:44 /scratch/data/data_dl1_run244_1_20190927T194408_dl2.root -rw-r--r-- 1 icarus P-1052 64688189 Sep 27 14:44 /scratch/data/data_dl1_run244_2_20190927T194411_dl2.root -rw-r--r-- 1 icarus P-1052 64688189 Sep 27 14:44 /scratch/data/data_dl1_run244_3_20190927T194414_dl2.root -rw-r--r-- 1 icarus P-1052 64688189 Sep 27 14:44 /scratch/data/data_dl1_run244_4_20190927T194416_dl2.root -rw-r--r-- 1 icarus P-1052 64688189 Sep 27 14:44 /scratch/data/data_dl1_run244_5_20190927T194418_dl2.root -rw-r--r-- 1 icarus P-1052 64688189 Sep 27 14:44 /scratch/data/data_dl1_run244_6_20190927T194421_dl2.root -rw-r--r-- 1 icarus P-1052 64688189 Sep 27 14:44 /scratch/data/data_dl1_run244_7_20190927T194423_dl2.root -rw-r--r-- 1 icarus P-1052 64688189 Sep 27 14:44 /scratch/data/data_dl1_run244_8_20190927T194426_dl2.root -rw-r--r-- 1 icarus P-1052 6778961 Sep 27 14:44 /scratch/data/data_dl1_run244_9_20190927T194428_dl2.root ============================================================================================== =======================================icarus-vst02======================================= ls: cannot access /scratch/sbndaq_eb00_r000244_sr*_*_*_dl1.root: No such file or directory The command "ssh icarus-vst02 'ls -l /scratch/sbndaq_eb00_r000244_sr*_*_*_dl1.root' " returned nonzero; this almost certainly means you won't get the info you want about the root file, if any, written by the EventBuilder1 process on icarus-vst02 for run 244. ==============================================================================================
The bottom part of the output seems to indicate that root files produced by EventBuilder1 can not be accessed or are missing; however, in reality, only DataLogger1 was producing root files during the run 244.
- Testing the "show_rootfiles_for_run.sh 244 5" script.
[icarus@icarus-vst01 DAQInterface]$ show_rootfiles_for_run.sh 244 5 Run number is 244 Will show the first 5 events from the root file(s) (if any) produced in the run =======================================icarus-vst02======================================= An alias for rawEventDump is not found in /home/nfs/icarus/run_records/244/setup.txt, will instead run a generic, experiment-independent version of the command ============================================================================================== =======================================icarus-vst02======================================= An alias for rawEventDump is not found in /home/nfs/icarus/run_records/244/setup.txt, will instead run a generic, experiment-independent version of the command ==============================================================================================
For some reason, the output does not show the list of root files, perhaps this is the desired behavior.
#5 Updated by John Freeman 8 months ago
The previous reviewer comments are addressed at the head of feature/23262_root_file_info (commit 26fdc095ee0e6e9a2560303f3487288ad35fb7f5). To elaborate a bit on what's been done:
- The fact that show_rootfiles_for_run.sh expected the eventbuilder to have produced a root file in the reviewer's test even though the RootOutput module wasn't in the my_output_modules list was the result of a silly bug where a line was left out of the script. Fixed.
- Concerning the root file not being listed when "show_rootfiles_for_run.sh 244 5" was executed: this was not desirable behavior, and has also been fixed.
- Also not desirable is that individual events weren't even listed via the generic rawEventDump when "show_rootfiles_for_run.sh 244 5" was run. I investigated, and the reason for this is that source-ing the DAQ setup script for that run as saved in the run record is how DAQInterface sets up art in order to list the events...but the source was resulting in an error. I've added more information for the user for when this particular failure mode occurs, including outputting the contents of the DAQ setup script source to screen.