Project

General

Profile

Feature #23262

A convenience script could be provided in DAQInterface to tell users where their root files are

Added by John Freeman 2 months ago. Updated about 1 month ago.

Status:
Reviewed
Priority:
Normal
Assignee:
Category:
-
Target version:
-
Start date:
09/11/2019
Due date:
% Done:

100%

Estimated time:
Experiment:
-
Co-Assignees:
Duration:

Description

DAQInterface currently has a couple of convenience scripts, documented in its manual, which help users look at the output of their run (e.g., "show_logfile_for_process.sh", which will point users to an artdaq process's logfile given the run number and process label). It might be nice to have a script which lists the output root files from a run, so users won't need to scrounge around in the saved eventbuilder or datalogger FHiCL documents to figure out what directory the root files were written to (and then have to look in the boot file to figure out which hosts the eventbuilders and dataloggers were on!). E.g., in the case of my personal run 2898 on the mu2edaq cluster

show_rootfiles_for_run.sh 2898

could return something like
mu2edaq01:/home/jcfree/daqdata/artdaqdemo_r002898_sr01_20190911T184200_1.root

Getting ambitious, the script could even do things like tell you if it expected a root file but didn't find one (e.g., if there were no events in the run, or a datalogger timed out or crashed), as well as tell you how many events were in the root file. Perhaps it could even call rawEventDump? Suggestions welcome.

Associated revisions

Revision 594116d7 (diff)
Added by John Freeman about 2 months ago

JCF: Issue #23262: allow more than one module to write root files in a process as long as only one is used

Revision 0a28daad (diff)
Added by John Freeman about 2 months ago

JCF: Issue #23262: be more robust as to the formatting the run number in the root filename can take

Revision 494df477 (diff)
Added by John Freeman about 2 months ago

JCF: Issue #23262: have show_rootfiles_for_run.sh fall back on a generic rawEventDump if no-other's available

Revision 566d69fa (diff)
Added by John Freeman about 2 months ago

JCF: Issue #23262: needed to add rawEventDump.fcl for the previous commit for this issue to work

Revision 10b0a0ef (diff)
Added by John Freeman about 1 month ago

JCF: Issue #23262: make sure show_rootfiles_for_run doesn't expect root files from unused modules

Revision 26fdc095 (diff)
Added by John Freeman about 1 month ago

JCF: Issue #23262: make sure that the root file is always listed whether or not users want to see event-by-event info

History

#1 Updated by John Freeman about 2 months ago

  • % Done changed from 0 to 100
  • Status changed from New to Resolved

At the head of feature/23262_root_file_info (c8d28363925d8e8def1fcd65a3f143831f1ba802) the show_rootfiles_for_run.sh script is ready for review. The simplest usage is to just pass it the run number:

show_rootfiles_for_run.sh <run number>

where it will then list the *.root file(s) produced during the run.

For a little more sophistication, you can enter a second, optional argument, which is the # of events you want rawEventDump to print out from each root file:

show_rootfiles_for_run.sh <run number> <# of events to list>

To experience the full capabilities of show_rootfiles_for_run.sh, it's a good idea to test it on a run which had multiple diskwriting artdaq processes spread across different nodes. Please note that if "rawEventDump" isn't already available in the shell AND there's no rawEventDump alias in the DAQ setup script for the run in question, show_rootfiles_for_run.sh will be unable to list the events, even though of course it can still list the root files.

#2 Updated by John Freeman about 2 months ago

Gennadiy played around with the script and had some useful insights into how it could be improved:

  • Right now, show_rootfiles_for_run.sh, when asked to display one or more events per file, relies on rawEventDump either already being available in the environment or being something that gets alias-ed in the DAQ setup script for the run in question. Would it be possible for it to have a bare-bones, experiment-independent default to fall back on if rawEventDump weren't available in either of those two aforementioned locations?
  • show_rootfiles_for_run.sh, when figuring out where to find root files, should have a broader knowledge of the formatting that can be used for filenames in the output modules. E.g., it can understand myfile_%06r.root, but not myfile_%R.root.
  • show_rootfiles_for_run.sh currently assumes that if a string of the form "filename: <file that ends in root>" appears in a process's FHiCL document, that it implies that root file gets written. However, this isn't a guarantee - e.g, you could have a RootOutput module which contains this line, but then the module doesn't appear in the my_output_modules path

#3 Updated by John Freeman about 2 months ago

With commit 566d69fa3d6c10c1dbc8b2ec2529e1059ee13945 at the head of feature/23262_root_file_info, the improvements described in the previous comment have been implemented, and this issue is once again ready for review.

#4 Updated by Gennadiy Lukhanin about 2 months ago

Testing this feature branch on icarus-vst01 as icarus.

The artdaq_daqinterface package was built using buildtool and installed into the /software/products_dev directory on icarus-vst01.

[icarus@icarus-vst01 DAQInterface]$ ups active |grep artdaq_daqinterface
artdaq_daqinterface  v3_06_01_FB23262 -f NULL                                    -z /software/products_dev

  1. Running the "show_recent_runs.sh 5" command.
    [icarus@icarus-vst01 DAQInterface]$ show_recent_runs.sh 5
    Run 240 (Sep 27 14:10) : standard                      : pmtx01
    Run 241 (Sep 27 14:18) : standard                      : pmtx01
    Run 242 (Sep 27 14:22) : standard                      : pmtx01
    Run 243 (Sep 27 14:37) : standard                      : pmtx01
    Run 244 (Sep 27 14:44) : standard                      : pmtx01
    
  2. Testing the "show_rootfiles_for_run.sh 244" script.
    [icarus@icarus-vst01 DAQInterface]$ show_rootfiles_for_run.sh 244
    Run number is 244
    Will show the first 0 events from the root file(s) (if any) produced in the run
    
    =======================================icarus-vst02=======================================
    -rw-r--r-- 1 icarus P-1052 64688189 Sep 27 14:44 /scratch/data/data_dl1_run244_1_20190927T194408_dl2.root
    -rw-r--r-- 1 icarus P-1052 64688189 Sep 27 14:44 /scratch/data/data_dl1_run244_2_20190927T194411_dl2.root
    -rw-r--r-- 1 icarus P-1052 64688189 Sep 27 14:44 /scratch/data/data_dl1_run244_3_20190927T194414_dl2.root
    -rw-r--r-- 1 icarus P-1052 64688189 Sep 27 14:44 /scratch/data/data_dl1_run244_4_20190927T194416_dl2.root
    -rw-r--r-- 1 icarus P-1052 64688189 Sep 27 14:44 /scratch/data/data_dl1_run244_5_20190927T194418_dl2.root
    -rw-r--r-- 1 icarus P-1052 64688189 Sep 27 14:44 /scratch/data/data_dl1_run244_6_20190927T194421_dl2.root
    -rw-r--r-- 1 icarus P-1052 64688189 Sep 27 14:44 /scratch/data/data_dl1_run244_7_20190927T194423_dl2.root
    -rw-r--r-- 1 icarus P-1052 64688189 Sep 27 14:44 /scratch/data/data_dl1_run244_8_20190927T194426_dl2.root
    -rw-r--r-- 1 icarus P-1052  6778961 Sep 27 14:44 /scratch/data/data_dl1_run244_9_20190927T194428_dl2.root
    ==============================================================================================
    
    =======================================icarus-vst02=======================================
    ls: cannot access /scratch/sbndaq_eb00_r000244_sr*_*_*_dl1.root: No such file or directory
    
    The command
    "ssh icarus-vst02 'ls -l /scratch/sbndaq_eb00_r000244_sr*_*_*_dl1.root' " 
    returned nonzero; this almost certainly means you
    won't get the info you want about the root file, if any, written by
    the EventBuilder1 process on icarus-vst02 for run 244.
    
    ==============================================================================================
    

    The bottom part of the output seems to indicate that root files produced by EventBuilder1 can not be accessed or are missing; however, in reality, only DataLogger1 was producing root files during the run 244.
  3. Testing the "show_rootfiles_for_run.sh 244 5" script.
    [icarus@icarus-vst01 DAQInterface]$ show_rootfiles_for_run.sh 244 5
    Run number is 244
    Will show the first 5 events from the root file(s) (if any) produced in the run
    
    =======================================icarus-vst02=======================================
    An alias for rawEventDump is not found in /home/nfs/icarus/run_records/244/setup.txt, will instead run a generic, experiment-independent version of the command
    ==============================================================================================
    
    =======================================icarus-vst02=======================================
    An alias for rawEventDump is not found in /home/nfs/icarus/run_records/244/setup.txt, will instead run a generic, experiment-independent version of the command
    ==============================================================================================
    

    For some reason, the output does not show the list of root files, perhaps this is the desired behavior.

#5 Updated by John Freeman about 1 month ago

The previous reviewer comments are addressed at the head of feature/23262_root_file_info (commit 26fdc095ee0e6e9a2560303f3487288ad35fb7f5). To elaborate a bit on what's been done:

  • The fact that show_rootfiles_for_run.sh expected the eventbuilder to have produced a root file in the reviewer's test even though the RootOutput module wasn't in the my_output_modules list was the result of a silly bug where a line was left out of the script. Fixed.
  • Concerning the root file not being listed when "show_rootfiles_for_run.sh 244 5" was executed: this was not desirable behavior, and has also been fixed.
  • Also not desirable is that individual events weren't even listed via the generic rawEventDump when "show_rootfiles_for_run.sh 244 5" was run. I investigated, and the reason for this is that source-ing the DAQ setup script for that run as saved in the run record is how DAQInterface sets up art in order to list the events...but the source was resulting in an error. I've added more information for the user for when this particular failure mode occurs, including outputting the contents of the DAQ setup script source to screen.

#6 Updated by John Freeman about 1 month ago

  • Status changed from Resolved to Reviewed

Reviewed by Gennadiy around lunchtime in our office today.



Also available in: Atom PDF