Detector and Beamline Reconstruction

Below are some details on how to run the full beamline plus detector reconstruction jobs on the grid. The output files for all these jobs are currently going to /pnfs/nova/scratch/users/tlackey/TBReco/. hadd-ed versions of the hist files exist in /nova/ana/users/tlackey/testbeam/analysis/tbrecoana. If we need access to the fully reconstructed files for longer than they'll stay on scratch, we need to discuss moving these to persistent.

Grid jobs

  1. Create a definition from the artdaq files you wish to process.
    samweb create-definition <def_name> "Online.Detector TestBeam AND Online.Stream Beamline AND data_tier artdaq AND run_number >= <run_low> AND run_number <= <run_high>" 
    You may need to add the raw2root release to the definition to avoid duplicates from times when we switched to a new release and double-processed files.
  2. Edit config script.
    Here's a sample config:
    --jobname lackey32_20200120_reco_r100500-509
    --defname lackey32_testbeam_artdaq_r100500-509
    --config beam_and_det_reco_plus_ana_job.fcl
    --tag S19-12-11
    --dest /pnfs/nova/scratch/users/tlackey/TBReco/r100500-509
    #only need a testrel if you have some local changes you want to run with
    --testrel /nova/app/users/tlackey/TestBeam/gridReco.S19-12-11
    --njobs 35
    --dynamic_lifetime 21600
    --outTier outreco:full-reconstructed
    --histTier hist
    --os SL6
    -G nova

    For each new submission, you'll want to makes changes to:
    • jobname
    • defname (to match the definition you create above)
    • dest (be sure to also create this output directory - alternatively you could send all jobs to the same output directory)
    • njobs (change based on the number of files in the definition - if the number is comparable to the last job, no change is necessary here)
  3. Run test submission. -f Reco_TB_beamline.cfg --test_submission
  4. If test looks good, submit the job. -f Reco_TB_beamline.cfg

Making plots

You can use TestBeamCommissioning/macros/tbrecoplots.C (or your own variation of this macro) to make plots of the data. If you're doing keepup processing to verify data quality each week, you'll want to update the t1 and nday variables. Make sure you edit the plotting macro to read in all the files you need and include an output directory in your own area to store the plots. The macro can be run in batch mode with:

root -b -q TestBeamCommissioning/macros/tbrecoplots.C