Project

General

Profile

Production 4 (Neutrino 2018 analysis campaign) Reconstruction book keeping

A record of the new/improved features in the production 4 processing campaign, production philosophy, and bookkeeping of software release history and relevant datasets.

Release notes up to date as of 1/05/18

Incremental production philosophy

The production 4 processing campaign for the Neutrino 2018 analysis follows an incremental model. The simulation and calibration is unchanged from the production 3 campaign. The starting point for production 4 is the production 3 PID level files. Only new or modified reconstruction products will be added on top of the existing reconstruction files. This strategy was adopted to save processing time. What follows is a description of the new products that exist in production 4 files. Due to the unique processing structure there are special FCLs which are also detailed later. It is important to note that to preserve the incremental nature of the processing the versions channel masks, geometry, and calibration constants must be identical between the production 3 processing and production 4.

What's changed in production 4 files

CVN

A number of training strategies were considered for the production 4 campaign. Two new CVN versions were evaluated in the mini-production campaign and compared to the version of CVN used in production rounds 2 and 3. The CAF structure was modified to accommodate three versions of CVN. Due the the incremental nature of the processing the production 3 version of CVN comes along for free in all files. Comparisons of CVN training versions and the choices made for production 4 processing are outlined in docdb-24729

  • CVN Short-Simple: This version of CVN is being run in the production 4 campaign. It is run under the module label cvnevalprod3train and is located in CAFs under the branch cvnProd3Train. This version of CVN was trained on production 3 MC files and cosmic data passing a loose preselection. This network has separate training for FHC and RHC. CVN identifies the network version to run based on the horn current information in the SpillData product, the module will abort without this product, ensure that the current stored in the product is correct. This version of CVN has 392 final state label outputs. In the CAFs Nue, Numu, Nutau, and NC summed outputs are still handled correctly. Note that the Cosmic output is number 390. *This CVN requires the CVNMapper to also be run before it. Due to a change in the pixel maps, the existing production 3 version of the product cannot be used. CVNMapper is run in production 4 under the label cvnmapprod4 *
  • CVN Classic: This version of CVN is the existing data product from the production 3 files. It was run in the previous production under the module label cvneval. Note that while this product was located in the CAF branch cvn in the production 3 files, it has been moved to cvn2017 in the production 4 files. This version of CVN has 16 output labels. The cosmic output is stored in number 14. Remember, this version of CVN is not being run again in production 4 processing, the existing data product will be copied straight to the CAFs.
  • CVN ELU: A branch for this CVN version exists in the CAFs but it has been deprecated. In production 4 CAFs this product exists under the branch cvn and was run under the module cvnevalprod4. It also has 392 final state output labels and was trained on production 2 FHC MC. After mini-production it was decided not to run this version of CVN in production 4, but not to change the CAF structure since the ND Data files did not need to be made again. In CAFs in production 4 other then ND Data the CAF branch will be filled with -5.

Regression CVN

A version of CVN that estimates the energy of interactions. Its was trained for Nue and electron energy estimation. This is a new product for the production 4 campaign. It is run under the two modules stages regcvnmap and regcvneval. In CAFs this product is stored under the FuzzyK prong branch in recon with prong and neutrino energy estimates associated with each prong.

FuzzyKVertex

This product was updated for production 4. A new orphaned hit object was added to the CAFs to account for the energy of hits in the slice that are not associated with any FuzzyKProng. The 2D prong energies have also been changed so that the w coordinate for the other view is the vertex coordinate rather then always being the center of the detector in the opposite view. This leads to a more accurate energy estimation for the 2D prongs. Finally, for all prongs a weighted energy is now produced to account for hits that are shared between prongs. The FuzzyK module was rerun in production 4 under the module label fuzzykvertexprod4. The weighted energy is stored in CAF for the prong under the variable weightedCalE, while the unweighted energy remains available under calE as it was in production 3.

NumuEnergy

The NumuEnergy estimator was updated based on the production 3 training and was trained separately for FHC and RHC. This product, like CVN, relies on the SpillData object horn current information to use the correct energy estimator. At the time of production 4 processing this is the state-of-the-art version of the enegy estimator, it is stored in the CAFs in the same format as in production 3 and replaces the old product.

BreakPointFitter

Updated algorithm with new dE/dx tables and improved tracking at start/endpoints. Data products are stored as they were in production 3 files.

TrackCleanup

Break out a new module, TrackOverlapECalc to run the TrackCleanupAlg on all 2D and 3D Kalmen and breakpoint fitter tracks to calculate the overlapping hadronic energy fraction. This had previously only been calculated for the leading (highest remid value) kalman track. In CAFs an overlapE variable was added to SRTracks.

NC CosRej

New module for cosmic rejection for the NC Analysis. In production 3 files this analysis was done at the CAFAna level. This module is run under the label nccosrej.

Numu CosRej

This module is rerun in production 4 to pick up BDT versions with CVN ShortSimple. Note, the BDT remains trained only over CVN Classic. A variable was added to the data product where the BDT is run with CVN ShortSimple as input. In CAFs the BDT using the CVN classic input is found under the variable numucontpidoldcvn now while the version with CVN ShortSimple as input is now under the variable numucontpid.

Additionally the numu CosRej data product and CAFs contain a version of the cosmic BDT designed for uncontained events.

The two numu cosrej variables run in {mini}prod4 are:

  • sr->sel.cosrej.numucontpidoldcvn: the old version used in prod3 which was trained using the cvn2017 cosmic score as input, which was only ever trained on FHC, simply rerun with the old training
  • sr->sel.cosrej.numucontpid: same prod3 version of the BDT training. NOTE: at evaluation time, it takes the newly trained CVN (cvnProd3Train) cosmic score as input (not the one it was initially trained with.)

Note: There was an error in processing the miniprod4 files in numu cosrej. The art-level code for the numu cosrej module hadn't been updated for the CVN final state labels. So the code was taking in the CVN score for label "14" (whatever that one is - can't recall off the top of my head, but it was one of the final state labels, not the correct cosmic label) instead of the correct cosmic score (label "390.") So this one is probably somewhat right (since the CVN score was only one input of many) but not entirely right. No studies have been done to determine which one is "less wrong."

This only affects the miniprod4 files. It has been corrected for the official prod4 files.

True uncontained energy

A new module to track and store truth quantities for the uncontained energy of particles exiting the detector.

NC Pi0 PID

New BDT developed for NC Pi0 event identification. Module is run under the label ncpi0bkgrej.

IFDBSpillInfo

CVN and the Numu energy estimator were trained separately for FHC and RHC. In beam files the correct version of the estimator is accessed using horn current information in the SpillData object. Cosmic files do no have horn information by default. The IFDBSpillInfo module was modified to have a special FCL setting, CosmicIsRHC to encode a horn current in cosmic files. It is set to RHC by default.

RewriteSpillDataMC

A new module introduced in production 4. It was discovered that in beam MC files the horn current was always set to FHC upon simulation generation in the SpillData object. This will be fixed at the source for future full productions. For the incremental production a temporary module was introduced to copy the existing SpillData object and copy in the correct horn current from the metadata. This module must be run in all production 4 jobs prior to any other reconstruction. Currently CVN and the numu energy estimator rely on a correct horn current setting to access the right training weights.

Important notes on calibration versions and bad channel masks

Summary of the problem w.r.t. incremental processing

It is important that when running production 4 processing on a production 3 PID file that the calibration constants, bad channel masks, and geometry remain unchanged. Otherwise the new/updated products are seeing a different version of the detector then the previous reconstruction, which proves particularly problematic for FuzzyK and CVN comparisons. The likely warning sign of a version skew will be a failure in CAFMaker when filling the FuzzyK information. This comes about since changes in the calibration/masks can produce different numbers of prongs then existed in production 3 processing. When CAFMaker is filling the new weighted prong energy information it requires that the number of prongs be the same. If you are doing your own processing, it is recommended to produce the initial PID file in a production 3 tagged release to ensure compatibility before moving to the incremental processing stage.

Notes on specific samples affected by this problem:

  • Note that for FD MC this means files must be produced in R17-03-01-prod3reco.l or later release. In particular, this makes the current production 3 CRY MC incompatible with production 4 processing and it will be reprocessed in a later release.
  • This problem was encountered again when processing period6 NuMI RHC data, which had been initially processed in the R17-03-01-prod3reco.k tag. Several commits were made ( r25928 , r26025 , r26026 , r26027 ) between the .k and .l tags related to channel masks and calibration constants for RHC data. All RHC data from period6 and period 7 will need to be processed in R17-03-01-prod3reco.l before proceeding to the prod4 jobs.
  • For all other varieties of files from the prod3 datasets (MC and data) the production 3 datafiles found on the production dataset webpage have been verified to be compatible with the incremental production.

How do I run production 4 incremental processing jobs?

Production 4 processing is designed to take a production 3 PID level file as input. Output ART files in production 4 are of the new tier repid to signify the additional objects added to the existing PID file.

For NuMI files and beam MC the standard processing job is prod_repid_numi_prod4_job.fcl

For cosmic data and CRY MC there are two separate processing jobs, run sequentially. CVN and the Numu energy estimator were trained separately for FHC and RHC. In beam files the correct version of the estimator is accessed using horn current information in the SpillData object. Cosmic files do no have horn information and require high statistics, so the cosmics are processed twice, once in each current configuration so both versions of the PID and energy estimator are available. The IFDBSpillInfo module has a special FCL setting to encode a horn current in cosmic files. It is set to RHC by default. The standard job also includes the MRBrem processing

  • In the second cosmic pass in FHC module use prod_repid_cvnfhc_cosmics_and_mrbrem_prod4_job.fcl where the input should be a repid level cosmic file in RHC mode. To reduce processing time only the products that care about horn current information are run a second time and the previous RHC versions of those products are dropped from the file.
  • Note that due to the nature of the incremental processing that some FCL parameters (such as the module labels in CAFMaker) differ from the standard configuration FCL. If you are spinning CAFs on your own or some piece of production 4 its recommended to review the configuration settings the modules were run under in the official production 4.

Where do I go for the most up-to-date file samples?

  • ND MC Mini-production sample valid for use before reprocessing happens to add stats: datasets
  • FD Data Processing in progress in R17-11-14-prod4reco.d: datasets
  • FD MC Processing complete in R17-11-14-prod4reco.d: datasets
  • FD Cosmic Data Processing complete in "RHC" mode in the R17-11-14-prod4reco.a tag: datasets
    The "FHC" pass is also complete, done in R17-11-14-prod4reco.b Note the FHC pass is a subset of the RHC pass, but matches within 1%.
  • FD CRY Not processed yet
  • Systematics Not processed yet

Notes on horn polarity in production 4 processing

In production 4 the polarity of the horn is determined by a voltage monitor, stored in the IFDB as E:HRNDIR and accessed through the IFDBSpillInfo module each spill. This monitor has a sampling rate and can occasionally miss the ramp in to horn and report an incorrect value. This is fine since the horn current monitors are very reliable and used in spill quality cuts for actual failures, they just don't have the sign of the polarity. For the next production we will switch to only reporting the polarity at the subrun level rather than spill-by-spill to avoid the appearance of a switch in current between spills. In reality it takes most a a day to switch polarity of the horns and would never happen between spills in a file.

There are two modules in reconstruction sensitive to the horn current. CVN and CAFAna both rely on the horn polarity information to load either the FHC or RHC version of the training. The training is loaded based only on the first event in the file, so subruns where the first event reported the wrong polarity will have the incorrect version of the training. If the incorrect polarity occurs mid file then the correct reconstruction training has still been applied, but there would be the chance of using the incorrect polarity information in CAFAna processes.

We plan to take the following actions for production 4:

  • In the FD data we plan to mark the following (run,subrun) combinations as bad where the first spill in the file had the incorrect horn polarity reported: (26151,07), (26151,08), (26218,35), (26675,56), (28233,26), (28333,30)
  • In the CAFAna spill cuts we plan to remove any individual spill that had the incorrect polarity so that any other cuts/vars that require polarity can be written in a future-proof way. This is estimated to only effect ~3,000 spills in FD RHC data. We will revisit other analysis level solutions if needed.

Technical notes on Production 4 Reconstruction

  • CVN TBA
  • Regression CVN TBA
  • FuzzyKVertex TBA
  • Breakpoint fitter TBA
  • True uncontained energy TBA
  • TDSlicer TBA

R17-09-05-prod4recopreview release series

For the purpose of running the 2018 analysis campaign mini-production processing. Series ended with the R17-09-05-prod4recopreview.f tag and moved to the production release branch as of 11/14/17. There are no outstanding bug fixes with this release series. Some varieties of files remain valid as part of the main analysis campaign and will be pointed out below.

R17-09-05-prod4recopreview.a

Additions/bugfixes

None, base release in series

Notes

deprecated release

Relevant datasets

None produced in this release

R17-09-05-prod4recopreview.b

Additions/bugfixes

  • r27224 Strip unused code in regression CVN looking for non-existent training files

Notes

deprecated release

Relevant datasets

None produced in this release

R17-09-05-prod4recopreview.c

Additions/bugfixes

  • r27238 Check vector size before filling

Notes

deprecated release

Relevant datasets

None produced in this release

R17-09-05-prod4recopreview.d

Additions/bugfixes

  • r27330
  • r27332 Change TrueEnergy verbosity and typo
  • r27339 Determine generator version correctly

Notes

deprecated release

Relevant datasets

None produced in this release

R17-09-05-prod4recopreview.e

Additions/bugfixes

  • r27398 Roll back bad channel FCL to replicate production 3 conditions

Notes

First release in series with bad channel masks that match production 3 conditions. 7 swapped channels in the FD had been added to the mask. This change lives on in development but cannot be brought into the production 4 branch due to the incremental nature of the processing since the existing data products did not see this mask.

Relevant datasets

Files were produced in this release, but only with CVN ELU and CVN Classic. The Short-Simple version of CVN had not been finalized yet. These files are superseded by those produced in the next release with all three CVN versions.

R17-09-05-prod4recopreview.f

Additions/bugfixes

  • r27579 Fix fill of spill data rewrite for the MC, MC POT not being stored properly in the CAFs before
Changes for Cosmic/MRBrem processing stream

Notes

Final release in mini-production processing series. POT information for MC properly stored in CAFs. All three CVN versions available in these files. Numu cosmic rejection with the new CVN is not pointing at the right input variable and should be disregarded. The version based on CVN classic is still valid

Known bugs:

There was an error in processing the miniprod4 files in numu cosrej. The art-level code for the numu cosrej module hadn't been updated for the CVN final state labels. So the code was taking in the CVN score for label "14" (whatever that one is - can't recall off the top of my head, but it was one of the final state labels, not the correct cosmic label) instead of the correct cosmic score (label "390.") So this one is probably somewhat right (since the CVN score was only one input of many) but not entirely right. No studies have been done to determine which one is "less wrong."

This only affects the miniprod4 files. It has been corrected for the official prod4 files.

Relevant datasets

  • ND Data files (both FHC and RHC) are considered complete and useable in this release. Use the ELU & SS CVN definitions here
  • ND MC files will be reproduced in a later production 4 release with full stats. This set of files is valid for studies in the meantime. Use the ELU & SS CVN definitions here
  • FD MC sample considered for selection of CVN algorithm. Full sets of MC being remade in a later production 4 release. Validation samples, Use the ELU & SS CVN definitions here

R17-11-14-prod4reco release series

Outstanding issues / planned features

none

R17-11-14-prod4reco.a

Additions/bugfixes

These items were added after the final round of mini-production files in R17-09-05-prod4recopreview.f were produced, but before the production 4 campaign began

Changes related to CVN
  • r28338 Reduce memory footprint, only load the FHC or RHC network version as needed
  • r28430 Set CVN ELU branch in CAFs to default -5 values
  • r28432 Update CVN UPS product to pick up CVN Short-Simple training
  • r28433 Make CVN Short-Simple the default FCL configuration option
  • r28484 Abort without horn current info, increase verbosity
  • r28572 Allow mapper modules to be selective in preselection, although currently not used
Changes related to NumuEnergy
  • r28470 Use horn current info to determine FHC/RHC instead of run numbers
  • r28488 Abort without horn current info
  • r28523 Increase horn current logging
  • r28578 Refactor fhc/rhc determination to remove duplicate information in the module and the algorithm
  • r28626 CAFAna part of the change
Changes related to IFDBSpillInfo
  • r28515 Add spill data product in cosmic files to store a horn current
  • r28555 Abort without horn current info, increase verbosity
Changes related to CosRej
  • r28445 CVN Short-Simple uses fine-grained labels, input to cosmic rejection is output 390
Changes related to FuzzyKVertex
  • r28445 CVN Short-Simple uses fine-grained labels, input to cosmic rejection is output 390
Changes related to TDSlicer
  • r28580 Fix bug where duplicate associations were being made
Changes related to special MC overlay samples Fill variable in Michel Finder

Notes

First release of the production 4 campaign.

Relevant datasets

FD Cosmic data in "RHC" mode processing in progress as of 11/26/17, datasets
The "FHC" pass of the cosmic data and other FD processing needs to be done in a later tag to pick up the fixed version of the uncontained numu BDT

R17-11-14-prod4reco.b

Additions/bugfixes

Change CVN version used in numu uncontained BDT from CVN ShortSimple back to CVN Classic.
  • r28754 Committed to prod4 branch
Pick up additional changes from production 3 branch to handle special MC overlay sample studies
  • r28591 avoid CAFMaker things that break in dataspill-mc overlay

Notes

Fixes bug for uncontained numu BDT mentioned in the .a release version. The release will be used for the "FHC" pass of FD cosmics and other processing goring forward

Relevant datasets

"FHC" pass of FD cosmics will be done in this release, to be updated when production starts

R17-11-14-prod4reco.c

Additions/bugfixes

The need to include "fIncompleteEvent" from the dq::DAQEventSummary art product.

  • As reported here the DAQ began running in a mode where previously fatal "undefined" buffer node error (causing the run to crash) is no longer fatal.
  • After running in this mode began, it was discovered that the buffer nodes when in this state (which previously crashed the run) appear to only be suffering a ~20 second interruption. But that during this interruption, the events on that buffer node are incomplete.
  • Running this way recovers lots of uptime, but we have to ensure that we can cut these events offline. This should be achievable by cutting on the "fIncompleteEvent" from the dq::DAQEventSummary art product. This has not yet been verified to work, but as of the writing of this entry (12/11/17) Justin V. was working on implementing this in CAFMaker.
    This has been addressed by the following commits:
    r29015 Add new DAQ header metrics to StandardRecord and CAFMaker
    r29016 Check for isEventIncomplete flag in kStandardDQCuts
    r29019 Fix a classes_def.xml boo boo
    r29020 Force CAFMaker to crash ("strict" mode) if dq::DAQEventSummary product is not present in the event
    r29022 Place DataQuality earlier in the build order (before CAFMaker)
    A couple notes about the behavior these changes produce:
    1. If any file does not have the dq::DAQEventSummary product, CAFMaker will abort when "strict" mode is enabled.
    2. MC files do not have a DAQ header. They still get a dq::DAQEventSummary product, but all of it's metrics are set to default values (0).
    3. The default behavior of the new cut is to do nothing. It only activates when the isEventIncomplete flag is explicitly set for that event in the DAQ header.
  • We will need to process at least all NuMI trigger data from period 7 with this fix in place. Still assessing the effect on earlier periods.

M121DS IFDB database beam width variable no longer available

  • r29034 This is not a variable we actually use in calculation of the beam width. But the IFDBSpillInfo module did read it in. Failing to grab it was causing data to fail spill related cuts. Removed this variable from the module. Will be picked up in period 7 data processing, earlier data is fine. The commit also made to the prod3reco branch so it can be applied in the first stage reconstruction.

Notes

Added CAF handles for "incomplete" events. At least required for period 7 data processing, investigation still ongoing.

Relevant datasets

None, passed over for R17-11-14-prod4reco.d to correct blinded CAFs

R17-11-14-prod4reco.d

Additions/bugfixes

Teach CAF Blinding about the new CVN, regression CVN, CVN Prong, and additional BPF and Kalman information that should have been in the blinded list.
  • "r29191:https://cdcvs.fnal.gov/redmine/projects/novaart/repository/revisions/29191
  • "r29191:https://cdcvs.fnal.gov/redmine/projects/novaart/repository/revisions/29191

Notes

Update blinded CAFs for FD NuMI data to know about new CVN and other more recent products that contain sensitive energy or PID information

For MC CAFs the isRHC variable was filled incorrectly. A new tag will be made to reCAF. The ART files and data CAFs are fine. The MC CAFs are useable if no cuts/vars dependent on horn current switching are used.

Relevant datasets

FD NuMI data, FD MC

R17-11-14-prod4reco.e

Additions/bugfixes

  • r29327 The isRHC variable was being filled incorrectly in CAFs for MC. Was looking at the wrong module for the information

Notes

ReCAFing FD/MC beam MC files in this release to fix isRHC information

Relevant datasets

RHC beam MC files

R17-11-14-prod4reco.f

Additions/bugfixes

  • r29379 Get metadata module to set HornPolarity fields in the case of non-beam MC (CRY).

Notes

Get metadata module to set the HornPolarity field for the special case of non-beam MC (CRY) where we use that field to keep track of CVN/NumuE tuning varieties even in the absence of actual beam.

Relevant datasets

At least used for Cosmic CRY processing to get the metadata set.

R17-11-14-prod4reco.g

Additions/bugfixes

  • r29517 Fix initialization issue effecting the first cell the query bad channels in the ND.

Notes

Rare ND failure mode where in the prod4 pass of reconstruction it was found that the first cell in the first event to query bad channels was always being flagged bad due to an initialization issue which was causing FuzzyK to occasionally produce different prongs then it had in the production 3 pass. Picked up for systematics processing.

Also, novaart:r30701 (mentioned below in the .h release as the branch commit novaart:r30702) was backported by hand into this release (as novaart:r30735) to simplify bookkeeping for FD data taking.

Relevant datasets

Picked up for systematic sample processing in ND.

Also used for some processing of FD NuMI data.

R17-11-14-prod4reco.h

Additions/bugfixes

  • r30702 Fixed "abort" in BreakPointFitter/func/Path.cxx to be "return" to allow the code to continue.

Notes

The new BPF dead material correction code had an abort in it in the case of extremely short tracks. It was expected that this abort would never be reached, but apparently some subtle cases slipped through when processing FD P1 NuMI data. The "abort" was changed to a "return" which causes the dead material correction factors for track {start, end} points to just skip over being calculated (resulting in just the "uncorrected" track that we would have had in prod3.)

Relevant datasets

Picked up for processing FD NuMI data. Actually the relevant commit here was backported in-place into the .g tag, so no data has been processed with this release. See R17-11-14-prod4reco.g above.

R17-11-14-prod4reco.remid-hotfix.b

Notes

It was discovered that in this fcl which was used to generate files used to make the ND neutron respin CAFs (among other things...) that BreakPointFitter was added with out rerunning the TrackOverlapECalc module in the NumuEnergy package. New BPF tracks were thus created and the associations with the OverlapE data products were broken. This resulted in the overlapE variable being set to -5 for BPF tracks in the ND CAF (but not for the Kalman tracks.) The FD neutron respin file appear to be fine since that respin only involved a remaking of CAFs with additional info (or so I am told...) This is intended to be fixed in the Fall-2018 CVN respin.

See also the slack conversations in the #bpf channel from 10/19/18 - 10-22/18 for more info.