Reproducing the nue Second Analysis results¶
- Table of contents
- Reproducing the nue Second Analysis results
Set up the appropriate release¶
- Setup the second analysis branch
setup_nova -b maxopt -r R16-07-15-secondana.a newrel -t R16-07-15-secondana.a mytestrel cd mytestrel srt_setup -a addpkg_svn -b CAFAna R16-07-15-secondana-br
Producing the prediction objects¶
The main object containing the nue prediction is PredictionSystNueSecondAna. It requires, among other things, a set of files which includes nominal and systematically shifted predictions, located in predDir
. By default, this directory is $NOVA_ANA/nu_e_ana/SecondAna/Predictions/CombinedPeriods/
.
If you need to recreate these files, but no interest in looking under the hood, just run
python $SRT_PUBLIC_CONTEXT/CAFAna/nue/SecondAna/make_systematics_all.py -pand wait ~ 1 day. Move the 157 output files to a folder of your choice. To learn more, keep reading.
Systematic prediction files¶
- Systematics for SA were run shift by shift in order to minimize memory issues at the BENDecomp step.
make_systematics_all.py
script runs all the samples in series in the following way: - Make predictions for 2 sets of periods/epochs (calls
make_nueFDprediction.C
). The analysis used 2 sets: 1+2+3b, and 3c. - Make combined predictions (calls
merge_predictions.C
) - Make plots, tables and systs templates (calls
plot_nueFDprediction_systematics.C
)
- Usage (
python make_systematics_all.py --help
)python make_systematics_all.py -n --nsets <number of samples to process> -f --first <sample to start at> -m --skipmake -a --skiphadd -c --skipcombine -p --skipplot
- List of samples: (order defined in
make_systematics_all.py
, datasets defined insystDatasets.h
). Note that the sample number required for--first
above would be 0-19 instead of 1-20 - Nominal
- Nominal - No Tau
- Calibration - Functional shift X
- Calibration - Functional shift Y
- Calibration - Scale shift positive
- Calibration - Scale shift negative
- Nominal - No rock
- Birks C
- Nominal - Genie Reweight
- Genie - MaCCQEshape
- Genie - NormCCQE
- Genie - MaCOH
- Genie - CCQEPauliSupViaKF
- Genie - MaRES
- Genie - MvRES
- Genie - allpi
- MEC Scale
- RPACCQE
- Beam - NA49
- Beam - All Transport
- The details of the Beam and Genie systematics evaluated is in
systVariations.h
. A few others that were not used in the final analysis are defined in the same file; they can be added to the list inmake_systematics_all.py
if needed.
- Also note that there are 4 Nominal predictions, to be matched with the different samples according to the details of the simulation available. Calibration samples need to be compared with a NoTau nominal; BirksC with NoRock; Genie with Genie. This information is saved in the
compareto
option inmake_systematics_all.py
and flags inana::DummyNueSyst
Examples¶
- Create only the nominal PredictionCombinePeriods:
python make_systematics_all.py -n1 -a -p
- Create the prediction files split by period in parallel, then combine, then hadd
- Run e.g. in different machines. ETA: ~6 hrs
python make_systematics_all.py -f0 -n5 -c -a -p python make_systematics_all.py -f5 -n5 -c -a -p python make_systematics_all.py -f10 -n5 -c -a -p python make_systematics_all.py -f15 -n5 -c -a -p
- Combine by periods. ETA: 30 min
python make_systematics_all.py -m -a -p
- Hadd different sigmas in a single file. ETA: 30 min
python make_systematics_all.py -m -c -p
- Run e.g. in different machines. ETA: ~6 hrs
- Different combination of periods
Everything is designed to be split in 2 sets, with a combination of periods/epochs from {p1,p2,p3b,p3c} (defaultperiods = ["p1p2p3b","p3c"]
inmake_systematics_all.py
). If you need two different sets, just modify that line (e.g.periods = ["p1p2","p3bp3c"]
). For different combinations, you'll need to callmake_
macro and create your ownPredictionCombinePeriods
elsewhere. For example, the lines below will create 3 predictions: p1, p2 and p3.cafe -bq make_nueFDprediction.C Nominal Nominal Nominal NueSA_prediction_Nominal_p1.root true 4 true & cafe -bq make_nueFDprediction.C Nominal Nominal Nominal NueSA_prediction_Nominal_p2.root true 4 true & cafe -bq make_nueFDprediction.C Nominal Nominal Nominal NueSA_prediction_Nominal_p3bp3c.root true 4 true &
Normalization systematics¶
In addition to the 16 systematics from the samples above, the analysis used 3 normalization systematics: Sig+Bkg; Sig; Bkg. These are set in PredictionSystNueSecondAna::AddNormSysts
.
Systematic plots, tables and templates¶
Producing the FD cosmic and data spectra¶
sa_box_opening_macro.C
makes in-time and out-of-time spectra for box opening. ETA 5 min
cafe -bq sa_box_opening_macro.C
The output file,
NueSA_numi_data.root
, contains a series of spectra named spec_{lid,lem,cvn}_{(pid)2d,nueE,time,slctime}_{numi,cosmic}
. Of special interest for the next steps in the analysis are spec_cvn_cvn2d_numi
and spec_cvn_cvn2d_cosmic
. This macro will also produce 6 text files (3 pids, numi or cosmic) with the values of &kRun, &kSubrun, &kEvt, &kSlice, &kSliceMeanTime, &kLID, &kLEM, &kCVNe, &kNueEnergy
for the selected events.
You can find copies of these outputs in /nova/ana/nu_e_ana/SecondAna
.
Event list and event displays¶
(Copied from commit message. TODO check, run, copy output to nu_e_ana (?))
make_sa_event_list
macro makes three lists of events that pass all nue selection cuts, one for each PID.
we would supply that as an input to the make_artfile_from_eventlist.sh
which will create an art file for event display scanning (needs a PID dataset). It will also run sa_box_opening_decaf
macro on the corresponding caf files, which will contain only events that pass any of the PIDs in the beam spill window.
Reproducing Data/MC comparisons¶
Near Detector Data/MC¶
Far Detector Data/MC¶
Reproducing the fit results¶
There are at least 3 distinct versions of the fit results:
- Nue only: using the nue data and prediction, and the PDG 2015 dmsq and reactor constraint
- Numu constraint: as presented in Neutrino 2016, in addition to the nue data and the reactor constraint, we use the numu result in the form of 2D chisq surfaces in ssth23 and dmsq32. This is not a joint fit!
- Nue+Numu joint fit: as prepared for the nue SA paper, we use both nue and numu data and predictions, plus the reactor constraint, fitting simultaneously. Correlations among systematic uncertainties ARE included.
These will reproduce the figures in the paper:
cafe -bq joint_fit_make_experiments.C CVN combo #saves experiments once for faster loading cafe -bq joint_fit_contours.C CVN combo true true joint true cafe -bq joint_fit_contours.C CVN combo false true joint true #joint fit, syst, FC cafe -bq joint_fit_contours.C CVN combo true true nueOnly true cafe -bq joint_fit_contours.C CVN combo false true nueOnly true #nue only, syst, FC cafe -bq joint_fit_slices.C CVN combo true true false true cafe -bq joint_fit_slices.C CVN combo false true false true true true #joint slices, octant constraint, FC
If you want to make other blessed plots or want more info, keep reading.
Nue only (+ numu constraint) contours¶
The macro contours.C
prints the S plot (th13) or wave plot (th23).
- Usage:
cafe -bq contours.C PID decomp numuConstr fccorr th13plot
Options forPID
:CVN, LID, LEM
. Options for decomp:combo
,prop
.
For the wave plots, only the CVN combo versions will have filled areas. Other PID/decomp combinations will print lines only.
numuConstr
toggles between the PDG dmsq constraint or the NOvA numu ssth23 x dmsq constraint.
fccorr=true
will use the FC Surfaces available.
- S plot: nue + PDG dmsq constraint
dCP vs ss2th13 contour. Prints 1 file, NH top, IH bottom.cafe -bq contours.C CVN combo false false true
- Wave plot: nue + reactor + PDG dmsq constraint
ssth23 vs dCP contour. Prints 2 files: NH, IHcafe -bq contours.C CVN combo
- Wave plot: nue + reactor + PDG dmsq constraint + Feldman-Cousins corrections
ssth23 vs dCP contour. Feldman-Cousins surfaces from/nova/ana/nu_e_ana/SecondAna/FC/ForPaper/FCWave_?H.root
; using 1,2,3 sigma _smooth versions.
Prints 2 files: NH, IH.cafe -bq contours.C CVN combo false true
- Wave plot: nue + reactor + numu constraint
ssth23 vs dCP contour. Numu 2D chisq surfaces provided by the numu group via email, renamed and placed in/nova/ana/nu_e_ana/SecondAna/SAoutputs_Numu.root
. FC corrections are not available.
Prints 2 files: NH, IH.cafe -bq contours.C CVN combo true
Joint fit contours¶
- First get a file with the nue and numu experiments:
cafe -bq joint_fit_make_experiments.C CVN combo
- Run the contours + plot
cafe -bq joint_fit_contours(PIDStr, decomp, bool createFile, bool corrSysts, TString options="joint", bool fccorr, bool dmsqSurf)
PIDStr and decomp
will look for the experiment created above.createFile = true
runs the fit,=false
makes the plots.
SetcorrSysts=false
for stats only
options
arejoint, nueOnly, numuOnly
Setfccorr=true
to correct the significances when FC are available
dmsqSurf=true
adds the dmsq x th23 (numu-style) plot to the list; takes longer to run
- Wave plot: nue + numu + reactor constraint, stats only, no FC
cafe -bq joint_fit_contours.C CVN combo true cafe -bq joint_fit_contours.C CVN combo false
- Wave plot: nue + numu + reactor constraint, with correlated systematics
cafe -bq joint_fit_contours.C CVN combo true true joint #create file cafe -bq joint_fit_contours.C CVN combo false true joint false # plot no FC cafe -bq joint_fit_contours.C CVN combo false true joint true # plot with FC
Note: to get the 0.2-0.75 axis range, setzoomIn=true
injoint_fit_contours.C
- Same macro can also run nue or numu only (e.g. with systematics, no FC). Some functionality is redundant with
contours.C
:/cafe -bq joint_fit_contours.C CVN combo true true nueOnly cafe -bq joint_fit_contours.C CVN combo false true nueOnly cafe -bq joint_fit_contours.C CVN combo true true numuOnly cafe -bq joint_fit_contours.C CVN combo false true numuOnly
Joint fit slices¶
- Requires the same experiment file as the contours
- Usage
cafe -bq joint_fit_slices( PID, decomp = "prop", bool createFile, bool corrSysts, bool th23Fixed, bool octantConstr, bool fccorr, bool fourierFit)
createFile = true
runs the fit,=false
makes the plots
SetcorrSysts = false
for stats only.
th23Fixed = true
uses the numu best fit results for th23, does not fit th23
octantConstr = true
seeds from the numu best fit; fits th23 but stays in the same octant (no jump).
If both are false, it fits th23 normally.
fourierFit
will soften the FC-corrected IH curve, esp. important in region where significance reaches 4 sigma
- Slice for the paper, nue+numu joint fit with octant constraint, correlated systematics, and FC corrections
cafe -bq joint_fit_slices.C CVN combo false true false true cafe -bq joint_fit_slices.C CVN combo false true false true true true #joint slices, octant constraint, FC
FC¶
We ran several million FC pseudo-experiments to properly translate a deltachisquare into a significance. There are macros for running a job of a user-specified number of pseudo-experiments.
Each run only produces points in a user-specified bin, so that we may produce more FC points in certain regions where our analysis is more sensitive.
Running a FC job on a gpvm¶
Set up the secondana.a release
setup_nova -b maxopt -r R16-07-15-secondana.a
Navigate to a test release with CAFAna checked out. Navigate to the nue SecondAna folder
cd CAFAna/nue/SecondAna
Contour FC¶
Run
cafe -bq make_fc_surf.C <int NPts> <int bin> <bool NH> <int iteration>
where NPts specifies the number of FC experiments to generate with this job, bin is the analysis bin that you want to correct, NH is a bool for deciding which true hierarchy to throw experiments in, and iteration is a bookkeeping parameter, insuring multiple FC output files have unique, human-readable names.
Slice FC¶
Run
cafe -bq make_fc_slice.C <int NPts> <double sinsqth23> <int bin> <bool NH> <int iteration>
where NPts specifies the number of FC experiments to generate with this job, sinsqth23 specifies the value of th23 to use when throwing your pseudo-experiments, bin is the analysis bin that you want to correct, NH is a bool for deciding which true hierarchy to throw experiments in, and iteration is a bookkeeping parameter, insuring multiple FC output files have unique, human-readable names.
Running many FC experiments¶
Submit many instances of make_fc_{surf,slice}.C on a grid node. It will take a few kiloCPU-days on the OSG to generate a large enough sample of FC experiments to be useful. Since that's hard, I'll point people to the files we used, all files (appropriately named) in
/nova/ana/nu_e_ana/SecondAna/FC/ForPaper/
If you do want to generate an independent set of FC points, you will need to ifdh cp a few files over in your grid submission script:
/pnfs/nova/persistent/analysis/nue/nu_e_ana/SecondAna/NueSA_numi_data.root // Numu SA data file, used for cosmic spectrum /pnfs/nova/persistent/analysis/nue/nu_e_ana/SecondAna/Predictions/NueSA_prediction_reduced_CVN_Combo.root // File with nue prediction /pnfs/nova/users/dpershey/InputForFC/pred2_numu_sa_123b_3c.root // File with numu prediction
Then wait several days, and you'll have FC pseudo-experiments.