Systematics strategy

General comments

...on the framework and philosophy of our systematics treatment.

  • Keep usage of new data structures to a bare minimum by using existing CAFAna ones as much as possible (Nitish).
    • If possible reduce the number of "extra" headers/dependencies (Ashley).
  • Reduce the amount of options in the main plotting/generating prediction scripts (Nitish).
    • Won't be able to reduced much --> most options are needed at the moment. Could see if we pass more through option string (Ashley).
  • Greater flexibility while adding different analysis-specific things (Nitish).
    • e.g. more options for variables/axes and cuts to apply (Ashley).

Known bugs/inconveniences the current version of the framework (in the usage of the framework not bugs in the actual systematics treatment).

  • Incorrect NearDet plots
  • Clean up formatting for PID bin labels and key on FarDet plots
  • Numu loads RHC quantiles by default
    • At syst_variations.h, line 248, determines whether FHC or RHC from the analysis string. This is counter-intuitive as everywhere else we extract this from the option string (and there are checks in place). If you don't include "FHC" or "RHC" in the analysis string it will default to RHC numu quantile boundaries.
  • Get rid of warnings.
    • e.g. tex_tables_util.h:481:19: warning: expression result unused [-Wunused-value]
      • for delete temp_nom,temp_plu,temp_min;
    • Both are easy to fix
    • Also some string format warnings
  • Overwrites plots with FHC/RHC counterparts.
    • When running plot_prediction_systs some plots get overwritten if you run the script for their opposite-horn-current equivalents. This behaviour occurs inconsistently.
    • My (hack-y) solution has been to make separate directories to run FHC and RHC, but it should be easy to fix this.
  • Include all dependencies for building LaTex document in committed code or provide instructions on how to get them.
    • etoolbox.sty is the main offender here.
    • We could also use a Makefile to check for dependencies.

Proposed improvements the framework that do not require a significant deviation from the current structure/work-flow.

  • Add functionality for interpolation in systematics plots.
    • To be able to form apples-to-apples comparisons for systematics that don't have shifts with the same sigma value, we need to be able to interpolate the sigma shifts within the plotting part of the framework. This becomes notably relevant for the file based systematics, where we have single sigma value shifts (up and/or down). If we change the sigma value these should represent, we have no way of comparing 1-sigma to 1-sigma shifts with another systematic.
    • PredictionSystJoint201* already inherit from PredictionInterp, so it shouldn't be too difficult to interpolate. But it may require some reasonably significant changes to the way we extract MC components from the shifted predictions.
  • Decouple the LaTex source generation from the plotting script.
    • One of the main annoyances with the current framework is if you want to make the "big PDF" with any specific list of systematics (including the full document with all systematics), you have to run plot_prediction_systs to generate all the plots and tables, even if you have already generated some of them. For example if you are making the full document and you have changed/added a single systematic, you still have to re-make the plots for them all, to get the correct .tex file.
    • I'd like to split these processes so we can make the plots in groups and generate different versions of the document without having to re-make the plots and tables.

Significant overhaul

...of the framework through changes to philosophy, work-flow or interface.

  • Move most of the interface to python, all the input options parsed here. (Nitish)
    • Use python to generate a header file in-situ with all the analysis cuts, axes, loaders, extraps, systs etc.
    • Main scripts just read in this header and generate predictions or make plots.
    • Along this vein, I've used a collection of jupyter notebooks/python scripts for submitting systematically shifted prediction jobs and checking/merging the output prediction files. We could potentially incorporate this into the same python interface to streamline the "baby-sitting" part of the work-flow. (Ashley)
  • We should still keep the existing functionality to load through PredictionSystJoint201* for the official predictions, as the point of adding that in was so we know we can produce all the plots and tables for the exact predictions that are going into the fits. It would be great to be able to load any prediction file you want not necessarily going through PredictionSystJoin201* if people want to look at predictions for any analysis.
    • This would also be great for debugging a small number of systematics.