Planning for fitter comparisons » History » Version 2

« Previous - Version 2/4 (diff) - Next » - Current version
Constantinos Andreopoulos, 01/08/2019 09:31 AM

Planning for first SBN fitter comparisons - Q1/2019

This is a living document.
First draft produced during the SBN-OSC WG meeting on January 8th, 2018.

Authors: C. Andreopoulos, <add your name>


We now have at least 3 oscillation fitting frameworks (SBNfit, CAFAna, VALOR) that have implemented an SBN oscillation analysis.
As it was stated in the planning document for the activities of this WG, this is a very desirable situation: Using multiple fitters and independent implementations can help catch bugs, resolve confusion in the implementation of some of the SBN analysis inputs (eg systematics), and gauge the (sometimes unexpected) effects of different approximations or of different statistical procedures. The aim is that all oscillation fitters will implement the same SBN oscillation analysis using official inputs (MC tunes, event selections, physics and detector systematics) provided by the experts of each sub-domain, working within the corresponding SBN WGs.

The goals of this exercise is to reproduce the proposal-era sensitivities, using updated inputs and with all 3 new oscillation fitting frameworks and, while doing so, to:
1) start standardising the types and formats of deliverables of each WG (towards the SBN oscillation fit), and
2) start defining procedures and developing code to enable detailed comparisons between all SBN fitters.

Ideally, we should understand/resolve the origin of differences between the fitters within a simple SBN analysis framework, before adding substantial new complexity.

Summary of types of comparisons to perform

1) Comparisons of nominal event spectra and spectrum components
2) Comparisons of systematic variations to event spectra
3) Comparisons of SBN-driven systematic constraints
4) Systematic parameter pulls
5) Appearance, disappearance and combined analysis sensitivities
6) Fits to pseudo-data including well-motivated systematic variations

Common fitter configurations and inputs

MC production:

Event selections:

Simulated CR backgrounds:

Simulated dirt and other non-fiducial backgrounds:

Development of common selection codes:
Production of common analysis (sbncode) DSTs:

Interaction systematics:

Flux systematics:

Detector systematics and inter-detector correlations:

Detailed description of comparisons to perform:*

1) Comparisons of selected nue CC-like and numu CC-like spectra at the different detector sites.

2) Comparisons of spectral components of selected nue CC-like and numu CC-like samples based on some commonly agreed MC truth categorisation.

3) Comparisons of 1,2 and 3 sigma systematic error bands for selected nue CC-like and numu CC-like spectra, using the full a-priori error assignments, and showing both the combined error band and separate error bands for a) flux-related, b) neutrino interaction-related, and c) detector related uncertainties only.

4) Comparisons of the systematic parameter constraints coming from simulated SBND data.

5) As item 3, but using the constrained rather than the a-priori error.

6) Appearance-only, disappearance-only and combined fit sensitivities in a 3+1 framework, for a few sets of true oscillation parameter values (including no oscillation).

7) Fits to a small set of pseudo-data that include some well-motivated systematic variations, for a small number of sets of true oscillation parameter values (including no oscillation).