Project

General

Profile

History

Novaart ยป

Reproducing the nue Second Analysis results

Set up the appropriate release

  • Setup the second analysis branch
setup_nova -b maxopt -r R16-07-15-secondana.a
newrel -t R16-07-15-secondana.a mytestrel
cd mytestrel
srt_setup -a
addpkg_svn -b CAFAna R16-07-15-secondana-br

Producing the prediction objects

The main object containing the nue prediction is PredictionSystNueSecondAna. It requires, among other things, a set of files which includes nominal and systematically shifted predictions, located in predDir. By default, this directory is $NOVA_ANA/nu_e_ana/SecondAna/Predictions/CombinedPeriods/.

If you need to recreate these files, but no interest in looking under the hood, just run 

python $SRT_PUBLIC_CONTEXT/CAFAna/nue/SecondAna/make_systematics_all.py -p
and wait ~ 1 day. Move the 157 output files to a folder of your choice. To learn more, keep reading.

Systematic prediction files

  • Systematics for SA were run shift by shift in order to minimize memory issues at the BENDecomp step. make_systematics_all.py script runs all the samples in series in the following way:
    1. Make predictions for 2 sets of periods/epochs (calls make_nueFDprediction.C). The analysis used 2 sets: 1+2+3b, and 3c.
    2. Make combined predictions (calls merge_predictions.C)
    3. Make plots, tables and systs templates (calls plot_nueFDprediction_systematics.C)
  • Usage (python make_systematics_all.py --help)
    python make_systematics_all.py -n --nsets <number of samples to process>
                               -f --first <sample to start at>
                               -m --skipmake      -a --skiphadd
                               -c --skipcombine   -p --skipplot
  • List of samples: (order defined in make_systematics_all.py, datasets defined in systDatasets.h). Note that the sample number required for --first above would be 0-19 instead of 1-20
    1. Nominal
    2. Nominal - No Tau
    3. Calibration - Functional shift X
    4. Calibration - Functional shift Y
    5. Calibration - Scale shift positive
    6. Calibration - Scale shift negative
    7. Nominal - No rock
    8. Birks C
    9. Nominal - Genie Reweight
    10. Genie - MaCCQEshape
    11. Genie - NormCCQE
    12. Genie - MaCOH
    13. Genie - CCQEPauliSupViaKF
    14. Genie - MaRES
    15. Genie - MvRES
    16. Genie - allpi
    17. MEC Scale
    18. RPACCQE
    19. Beam - NA49
    20. Beam - All Transport
  • The details of the Beam and Genie systematics evaluated is in systVariations.h. A few others that were not used in the final analysis are defined in the same file; they can be added to the list in make_systematics_all.py if needed.
  • Also note that there are 4 Nominal predictions, to be matched with the different samples according to the details of the simulation available. Calibration samples need to be compared with a NoTau nominal; BirksC with NoRock; Genie with Genie. This information is saved in the compareto option in make_systematics_all.py and flags in ana::DummyNueSyst

Examples

  • Create only the nominal PredictionCombinePeriods:
    python make_systematics_all.py -n1 -a -p
  • Create the prediction files split by period in parallel, then combine, then hadd
    • Run e.g. in different machines. ETA: ~6 hrs
      python make_systematics_all.py -f0  -n5 -c -a -p
python make_systematics_all.py -f5  -n5 -c -a -p
python make_systematics_all.py -f10 -n5 -c -a -p
python make_systematics_all.py -f15 -n5 -c -a -p
    • Combine by periods. ETA: 30 min
      python make_systematics_all.py -m -a -p
    • Hadd different sigmas in a single file. ETA: 30 min
      python make_systematics_all.py -m -c -p
  • Different combination of periods
    Everything is designed to be split in 2 sets, with a combination of periods/epochs from {p1,p2,p3b,p3c} (default periods = ["p1p2p3b","p3c"] in make_systematics_all.py). If you need two different sets, just modify that line (e.g. periods = ["p1p2","p3bp3c"]). For different combinations, you'll need to call make_ macro and create your own PredictionCombinePeriods elsewhere. For example, the lines below will create 3 predictions: p1, p2 and p3.
    cafe -bq make_nueFDprediction.C Nominal Nominal Nominal NueSA_prediction_Nominal_p1.root true 4 true &
cafe -bq make_nueFDprediction.C Nominal Nominal Nominal NueSA_prediction_Nominal_p2.root true 4 true &
cafe -bq make_nueFDprediction.C Nominal Nominal Nominal NueSA_prediction_Nominal_p3bp3c.root true 4 true &

Normalization systematics

In addition to the 16 systematics from the samples above, the analysis used 3 normalization systematics: Sig+Bkg; Sig; Bkg. These are set in PredictionSystNueSecondAna::AddNormSysts.

Systematic plots, tables and templates

Producing the FD cosmic and data spectra

sa_box_opening_macro.C makes in-time and out-of-time spectra for box opening. ETA 5 min

cafe -bq sa_box_opening_macro.C

The output file, NueSA_numi_data.root, contains a series of spectra named spec_{lid,lem,cvn}_{(pid)2d,nueE,time,slctime}_{numi,cosmic}. Of special interest for the next steps in the analysis are spec_cvn_cvn2d_numi and spec_cvn_cvn2d_cosmic. This macro will also produce 6 text files (3 pids, numi or cosmic) with the values of &kRun, &kSubrun, &kEvt, &kSlice, &kSliceMeanTime, &kLID, &kLEM, &kCVNe, &kNueEnergy for the selected events.

You can find copies of these outputs in /nova/ana/nu_e_ana/SecondAna.

Event list and event displays

(Copied from commit message. TODO check, run, copy output to nu_e_ana (?))

make_sa_event_list macro makes three lists of events that pass all nue selection cuts, one for each PID.

we would supply that as an input to the make_artfile_from_eventlist.sh which will create an art file for event display scanning (needs a PID dataset). It will also run sa_box_opening_decaf macro on the corresponding caf files, which will contain only events that pass any of the PIDs in the beam spill window.

Reproducing Data/MC comparisons

Near Detector Data/MC

Far Detector Data/MC

Reproducing the fit results

There are at least 3 distinct versions of the fit results:

  • Nue only: using the nue data and prediction, and the PDG 2015 dmsq and reactor constraint
  • Numu constraint: as presented in Neutrino 2016, in addition to the nue data and the reactor constraint, we use the numu result in the form of 2D chisq surfaces in ssth23 and dmsq32. This is not a joint fit!
  • Nue+Numu joint fit: as prepared for the nue SA paper, we use both nue and numu data and predictions, plus the reactor constraint, fitting simultaneously. Correlations among systematic uncertainties ARE included.

These will reproduce the figures in the paper:

cafe -bq joint_fit_make_experiments.C CVN combo  #saves experiments once for faster loading

cafe -bq joint_fit_contours.C CVN combo true true joint true 
cafe -bq joint_fit_contours.C CVN combo false true joint true  #joint fit, syst, FC 

cafe -bq joint_fit_contours.C CVN combo true true nueOnly true 
cafe -bq joint_fit_contours.C CVN combo false true nueOnly true  #nue only, syst, FC 

cafe -bq joint_fit_slices.C CVN combo true true false true 
cafe -bq joint_fit_slices.C CVN combo false true false true true true #joint slices, octant constraint, FC

If you want to make other blessed plots or want more info, keep reading.

Nue only (+ numu constraint) contours

The macro contours.C prints the S plot (th13) or wave plot (th23).

  • Usage:
    cafe -bq contours.C PID decomp numuConstr fccorr th13plot

    Options for PID: CVN, LID, LEM. Options for decomp: combo, prop.
    For the wave plots, only the CVN combo versions will have filled areas. Other PID/decomp combinations will print lines only.
    numuConstr toggles between the PDG dmsq constraint or the NOvA numu ssth23 x dmsq constraint.
    fccorr=true will use the FC Surfaces available.
  • S plot: nue + PDG dmsq constraint
    dCP vs ss2th13 contour. Prints 1 file, NH top, IH bottom.
    cafe -bq contours.C CVN combo false false true
  • Wave plot: nue + reactor + PDG dmsq constraint
    ssth23 vs dCP contour. Prints 2 files: NH, IH
    cafe -bq contours.C CVN combo
  • Wave plot: nue + reactor + PDG dmsq constraint + Feldman-Cousins corrections
    ssth23 vs dCP contour. Feldman-Cousins surfaces from /nova/ana/nu_e_ana/SecondAna/FC/ForPaper/FCWave_?H.root; using 1,2,3 sigma _smooth versions.
    Prints 2 files: NH, IH. 
    cafe -bq contours.C CVN combo false true
  • Wave plot: nue + reactor + numu constraint
    ssth23 vs dCP contour. Numu 2D chisq surfaces provided by the numu group via email, renamed and placed in /nova/ana/nu_e_ana/SecondAna/SAoutputs_Numu.root. FC corrections are not available.
    Prints 2 files: NH, IH. 
    cafe -bq contours.C CVN combo true

Joint fit contours

  • First get a file with the nue and numu experiments:
    cafe -bq joint_fit_make_experiments.C CVN combo
  • Run the contours + plot
    cafe -bq joint_fit_contours(PIDStr, decomp, bool createFile, bool corrSysts, TString options="joint", bool fccorr, bool dmsqSurf)

    PIDStr and decomp will look for the experiment created above. createFile = true runs the fit, =false makes the plots.
    Set corrSysts=false for stats only
    options are joint, nueOnly, numuOnly
    Set fccorr=true to correct the significances when FC are available
    dmsqSurf=true adds the dmsq x th23 (numu-style) plot to the list; takes longer to run
  • Wave plot: nue + numu + reactor constraint, stats only, no FC
    cafe -bq joint_fit_contours.C CVN combo true
cafe -bq joint_fit_contours.C CVN combo false
  • Wave plot: nue + numu + reactor constraint, with correlated systematics
    cafe -bq joint_fit_contours.C CVN combo true  true joint #create file
cafe -bq joint_fit_contours.C CVN combo false true joint false # plot no FC
cafe -bq joint_fit_contours.C CVN combo false true joint true  # plot with FC

    Note: to get the 0.2-0.75 axis range, set zoomIn=true in joint_fit_contours.C
  • Same macro can also run nue or numu only (e.g. with systematics, no FC). Some functionality is redundant with contours.C :/
    cafe -bq joint_fit_contours.C CVN combo true  true nueOnly 
cafe -bq joint_fit_contours.C CVN combo false true nueOnly 

cafe -bq joint_fit_contours.C CVN combo true  true numuOnly 
cafe -bq joint_fit_contours.C CVN combo false true numuOnly

Joint fit slices

  • Requires the same experiment file as the contours
  • Usage
    cafe -bq joint_fit_slices( PID, decomp = "prop", bool createFile, bool corrSysts, bool th23Fixed, bool octantConstr,
                       bool fccorr, bool fourierFit)

    createFile = true runs the fit, =false makes the plots
    Set corrSysts = false for stats only.
    th23Fixed = true uses the numu best fit results for th23, does not fit th23
    octantConstr = true seeds from the numu best fit; fits th23 but stays in the same octant (no jump).
    If both are false, it fits th23 normally.
    fourierFit will soften the FC-corrected IH curve, esp. important in region where significance reaches 4 sigma
  • Slice for the paper, nue+numu joint fit with octant constraint, correlated systematics, and FC corrections
    cafe -bq joint_fit_slices.C CVN combo false true false true 
cafe -bq joint_fit_slices.C CVN combo false true false true true true #joint slices, octant constraint, FC

FC

We ran several million FC pseudo-experiments to properly translate a deltachisquare into a significance. There are macros for running a job of a user-specified number of pseudo-experiments.
Each run only produces points in a user-specified bin, so that we may produce more FC points in certain regions where our analysis is more sensitive.

Running a FC job on a gpvm

Set up the secondana.a release

setup_nova -b maxopt -r R16-07-15-secondana.a

Navigate to a test release with CAFAna checked out. Navigate to the nue SecondAna folder

cd CAFAna/nue/SecondAna

Contour FC

Run

cafe -bq make_fc_surf.C <int NPts> <int bin> <bool NH> <int iteration>

where NPts specifies the number of FC experiments to generate with this job, bin is the analysis bin that you want to correct, NH is a bool for deciding which true hierarchy to throw experiments in, and iteration is a bookkeeping parameter, insuring multiple FC output files have unique, human-readable names.

Slice FC

Run

cafe -bq make_fc_slice.C <int NPts> <double sinsqth23> <int bin> <bool NH> <int iteration>

where NPts specifies the number of FC experiments to generate with this job, sinsqth23 specifies the value of th23 to use when throwing your pseudo-experiments, bin is the analysis bin that you want to correct, NH is a bool for deciding which true hierarchy to throw experiments in, and iteration is a bookkeeping parameter, insuring multiple FC output files have unique, human-readable names.

Running many FC experiments

Submit many instances of make_fc_{surf,slice}.C on a grid node. It will take a few kiloCPU-days on the OSG to generate a large enough sample of FC experiments to be useful. Since that's hard, I'll point people to the files we used, all files (appropriately named) in

/nova/ana/nu_e_ana/SecondAna/FC/ForPaper/

If you do want to generate an independent set of FC points, you will need to ifdh cp a few files over in your grid submission script:

/pnfs/nova/persistent/analysis/nue/nu_e_ana/SecondAna/NueSA_numi_data.root // Numu SA data file, used for cosmic spectrum
/pnfs/nova/persistent/analysis/nue/nu_e_ana/SecondAna/Predictions/NueSA_prediction_reduced_CVN_Combo.root // File with nue prediction
/pnfs/nova/users/dpershey/InputForFC/pred2_numu_sa_123b_3c.root // File with numu prediction

Then wait several days, and you'll have FC pseudo-experiments.