Project

General

Profile

Calibration tags » History » Version 45

« Previous - Version 45/49 (diff) - Next » - Current version
Keith Matera, 03/01/2017 12:07 PM


Calibration tags

Official location of backup calibration CSV files in the case of database failures (starting Apr 25 2016):

/nova/ana/calibration/calibcsvs/

Official location of (outmoded) backup calibration CSV files (pre-Apr 25 2016; left in place for now in case anyone's code relies upon it):

/nova/ana/calibration/csvs/

Self-explanatory directory structure.

Details on how to create a calibration UPS product are given in this wiki page: calibration-tagging. Input files for calibration can be monitored here for artdaq files and here for pc-list files.

Latest full production versions (prod2)

FarDet NearDet NDOS
Data v9 v9 Legacy
MC v9 v9 Legacy

(Setting a newer tag falls back to an older version where necessary)

The results of latest attenuation fits are displayed for each channel on this webpage

Latest mini-production versions (prod3 miniproduction)

FarDet NearDet NDOS
Data v10 v10 Legacy
MC v10 v10 Legacy

Legacy (pre Validity Database)

  • Attenuation constants for NDOS data and MC, ND and FD MC
  • Absolute calibration hardcoded into Calibrator
  • MC calibrations are for untuned MC
  • No corrections for threshold effects applied
  • Exact datasets used lost in the mists of time

v1

  • Attenuation and absolute calibration constants for FD data and MC
  • No drift constants
  • MC calibrations are for untuned MC
  • Data calibration are for diblocks 1 and 2, from runs 11496-11555 (cold, high gain), using Kanika's good runs list
  • Threshold effect corrections included, but taken from untuned MC
  • Confusion around updated PE/ADC conversions leads to 20% shift in absolute scale

v2

v3

  • MC constants unchanged from v2
  • New attenuation and absolute calibration constants for FD data
  • Attenuation using corrections determined from v2 tuned MC
  • Constants for the first 4 diblocks based on data from run 13153 (Feb 15 2014) - 14029 (March 18 2014)

v3.1

Same as v3, but with "averaged" data attenuation constants included, for use in diblocks that are not yet calibrated.

v4

FD data

Data from runs 14592 (start of 10Hz triggering) to 15585 (last run before P1 has diblocks missing from the middle)

The processed attenprof files: S14-06-09_FD_data_attenprofs_for_v4

Absolute calibration and averaged constants carried over from previous versions

ND MC

Absolute energy scale determined "by hand" using reco/true of simulated cosmic ray hits because the production group are in a hurry.

v5

ND data

Data from runs 10408 (full cooled detector) to 10493 (most recent processed at time of calibration)

The processed attenprof files: S14-10-15_ND_data_attenprofs_for_v5

ND MC

The processed attenprof files: S14-09-29_ND_mc_attenprofs_for_v5

v5.1

Same as v5 except that ND data absolute constants are set to be those for (v5) ND MC. This should resolve the 9% shift between data and MC that's observed in the Near Detector. The underlying problem is the selection of a lot of showering events in the ND data (docdb 12435). This is only a temporary patch until that is fixed.

v6

ND data

Update to attenuation calibration only to fit cell-by-cell in the main body of the detector and to keep averages only for the muon catcher.

Data from runs 10408 (full cooled detector) to 10602 (most recent processed at time of calibration)

v7

Updates:
New cosmic muon selection for pchit files to improve the selection of cosmic muons dramatically.
New attenuation formula representing a small correction of the original.
New fit strategy for the muon catcher where fits are forced to converge using lowess.
FD MEU values found using Gaussian method before and mean now.

ND data

ND mc

FD data

FD mc

v7.1

Identical to v7 except the ND data absolute energy scale has been adjusted. Data hits will now be 3.7% higher in energy.

See docdb 13355 for the source of this number (+3.7%=1/0.964). Other calibration cross-checks show values scattered around a similar shift (docdb 13373 for an overview).

This is the calibration version used for the First Analysis.

v8

These are the calibrations produced prior to the second analysis or prod2. These calibrations are the first since a major re-tune of the simulated light levels in the detector. As a result the final calibrated energy for a given ADC signal has gone up by around 40%. Details can be found in docdb-14727 and references therein.

For the first time this calibration includes validity periods. That is to say two FD calibrations are provided, one for low gain (100) data/MC and a second for high gain (150) data/MC. As a result new naming conventions were brought in to differentiate between periods, as was new calibrator logic. This logic is explained on this page: calibration-tagging. The CSV files that match these calibrations are in the UPS product v08.02 (note this is not the same thing as a calibration tag - although the two are often tightly coupled). In the process of creating this product two earlier versions were made:

  • v08.00 - has the wrong FD data calibrations in and should not be used. * v08.01 - has a bug in the whitespace for the absolute calibration and should not be used.

When the dust has settled and we have a validated product these bad products will be removed.

This is the calibration used for the reco-preview files for prod 2.

Mini production datasets used are as following

ND data

ND MC

FD data (low gain)

FD data (high gain)

FD MC (low gain)

FD MC (high gain)

v9

These are the second analysis calibration files. They have all of the major changes included in v8. The difference with respect to v8 is that a simulation bug related to split tracks has been fixed.

FD Data

All Far Detector data are divided into different periods; period 1, 2 and 3 as
described in the Period and Epoch Naming production page. Here period
3 corresponds to the FD high gain era. The calibration has been performed separately for different periods.

Datasets used for the NOvA second analysis are,

Period 1: prod_pclist_R16-01-27-prod2calib.a_fd_cosmic_period1_quarter_goodruns
h4. Period 2: prod_pclist_R16-01-27-prod2calib.a_fd_cosmic_period2_quarter_goodruns
h4. Period 3 epoch 3b: prod_pclist_R16-01-27-prod2calib.a_fd_cosmic_epoch3b_goodruns
h4. Period 3 epoch 3c: prod_pclist_R16-01-27-prod2calib.a_fd_cosmic_epoch3c_goodruns

FD Monte Carlo

The Far Detector MC are divided into low gain and high gain era.
Datasets used for the FD MC calibration are,

Low gain: prod_pclist_R16-01-27-prod2calib.e_fd_cry_all_nova_v08_ideal_batch1_v1_gain100

High gain: prod_pclist_R16-01-27-prod2calib.e_fd_cry_all_nova_v08_ideal_batch1_v1_gain140

ND Data

As in the FD, the ND data are also divided into different periods, period 2 and 3. We do not use the period 1 for Near Detector calibration because there are not enough data in that period. We use the period 2 attenuation constants for period 1 runs. Datasets for different are,

Period 2: prod_pclist_R16-01-27-prod2calib.e_nd_ddactivity_period2_v2_goodruns

Period 3 epoch 3b: prod_pclist_R16-01-27-prod2calib.e_nd_ddactivity_epoch3b_v3_goodruns

Period 3 epoch 3c: prod_pclist_R16-01-27-prod2calib.e_nd_ddactivity_epoch3c_v2_goodruns

ND MC

A single dataset is used for ND MC second analysis calibration because of no changes in the Near Detector configurations. Dataset used for the ND MC calibration is,

prod_pclist_R16-01-27-prod2calib.e_none

v10 (currently: v10.01)

These are the calibrations produced prior to the third analysis or prod3. They were employed by the pre-prod3 miniproduction, which ran over the fileset defined here:

http://nusoft.fnal.gov/nova/production/miniprod2016/

Major players and conveners in the v10 calibration

What files can this calibration version be used for

  • ND MC files
  • ND data files
  • FD MC files (as of v10.01)
  • FD data files (as of v10.01)

What tags are associated with this calibration version

  • In Calibrator.fcl:
    • Tag: v10
    • AttenEpochTag: v2.5
    • AbsEpochTag: v2.5
    • TimingEpochTag: v2.5
    • TimingTag: tv4
  • In nova-offline-ups-externals-development, install_nova_externals_development, novasoft/ups/product_deps
    • Version: v10.01

Validity contexts

The following validity contexts were used to generate calibration constants. Each bolded phrase, separated from others by a comma, is a validity context. Each validity context has its own set of attenuation and absolute calibration constants.

FD MC: Full, Gain 100, and Gain 140 (two sets of constants)
ND MC: Full (one set of constants)

FD Data: Epoch2b, and Epoch 3b (two sets of constants)
ND Data: Period 1, Period 2, and Period 3 (three sets of constants)

The runs covered by each set of calibration constants are given below. In numbers of the form r#-r#, if the first # is missing, that means that the low run is 0 (the earliest possible run). If the second # is missing, that means that the high run is infinity (the latest possible run).

FD MC Gain 100: r-r20752
FD MC Gain 140: r20753-r,1,20753

ND MC: r-r

FD Data Epoch 2b: r-r20752
FD Data Epoch 3b: r20753-r

ND Data Period 2: r-r11228
ND Data Period 3: r11229-r

N.B. that we often extrapolate constants derived in one validity context to another (e.g., the FD Epoch2b constants are used to cover FD Periods 1 & 2, and the ND Period 2 constants are used to cover ND Periods 1 & 2). This is intentional. If you suspect that these validity contexts are being extrapolated wrongly, please let your calibration convener know.

Major changes introduced between v9 and v10

  • The geometry gdml files for both near and far detectors were changed. The biggest effect of this is to shrink the cell interior, increasing PE/cm by ~2% (docdb 15421)
  • Cherenkov light was added to the MC simulation (docdb 16349 as a starting point)
  • Fiber brightnesses were added to the MC simulation (docdb 15419 and docdb 16347 as a starting point)
  • MC files are calibrated by fiber brightness in the following fashion:
    • A single x and y plane of shadowing/threshold corrections are generated for each detector
    • An x and y plane of attenuation corrections is generated for each fiber brightness, for each detector (9 FB values x 2 views = 18 planes per detector)
    • The muon catcher is treated as its own detector for the purposes of shadowing/threshold and attenuation corrections, giving another 18 planes (bringing the total to 36 for the Near Detector)
    • There is now one atten*.csv file for each fiber brightness in the MC, e.g., from calib_atten_consts.nd.mc.v10.r-r.csv --> calib_atten_consts.nd.mc.v10.r-r.fb{0-8}.csv. This reflects that we now load a vector of AttenCache objects to apply the attenuation calibrations, each of which is represented by its own set of constants. See /grid/fermiapp/products/nova/externals/calibcsvs/v10.00/NULL/csv/v10/ for a full set of .csv files.
  • Data is calibrated in the same fashion as in earlier calibrations

Hacks, deviations, and exceptions to the above Major changes section

  • The muon catcher was calibrated using the v9 calibration constants. The v9 muon catcher attenuation constants files (just one, for each the x and y views, giving a total of 2 atten_consts and atten_points files) were copied over 9 times, and labeled according to the fb labeling described in the previous section (changing .csv --> .fb{0-8}.csv)

Changes between v10.00 (Dec 2016) and v10.01 (Jan 2016)

  • FD data calibration constants added, allowing for the calibration of FD data and FD MC files. (Why FD MC files only now? The FD MC cosmics overlay files use data hits, which must be calibrated using data calibration constants.)

Plans in the works for Feb 2017 and beyond

  • Make new Calibration/Calibrator code backwards compatible with pre-v10 calibration constants files
  • Update muon catcher calibration constants to use v10 constants rather than v9 constants
  • Reconsider using separate threshold/shadowing corrections for different fiber brightness values

v11 (currently v11.02)

These are the calibrations produced for the 2017 analysis production cycle, or prod3. They were employed by the prod3calib production cycle, which ran over the calibration samples defined here:

http://nusoft.fnal.gov/nova/production/miniprod2016/

Major players and conveners in the v11 calibration

What files can this calibration version be used for

  • ND MC files
  • ND data files
  • FD MC files
  • FD data files

What tags are associated with this calibration version

  • In Calibrator.fcl:
    • Tag: v11
    • AttenEpochTag: v2.6
    • AbsEpochTag: v2.6
    • TimingEpochTag: v2.5
    • TimingTag: tv4
  • In nova-offline-ups-externals-development, install_nova_externals_development, novasoft/ups/product_deps
    • Version: v11.02

Validity contexts

The following validity contexts were used to generate calibration constants. Each bolded phrase, separated from others by a comma, is a validity context. Each validity context has its own set of attenuation and absolute calibration constants. The runs covered by each set of calibration constants are provided after the colon. In numbers of the form r#-r#, if the first # is missing, that means that the low run is 0 (the earliest possible run). If the second # is missing, that means that the high run is infinity (the latest possible run).

  • ND Data Periods 1&2: r-r11228
  • ND Data Period 3: r11229-r11628
  • ND Data Period 4: r11632-r11925
  • ND Data Period 5: r11926-r
  • ND MC: r-r
  • FD Data Period 1&2: r-r20752
  • FD Data Period 3: r20753-r23419
  • FD Data Period 4: r23420-r24613
  • FD Data Period 5: r24614-r
  • FD MC Gain 100: r-r20752
  • FD MC Gain 140: r20753-r

Each validity context defined above uses all events which are part of the calibration samples defined here: http://nusoft.fnal.gov/nova/production/miniprod2016/ , and which also that occur within that context. This calibration process was not statistics-limited -- if we use less than a full period's worth of a dataset, it is because we only needed that many events. We then extrapolate to cover the full period, assuming consistency in detector behavior. If you disagree with a particular use of this extrapolation, please contact a Calibration convener with specific references (pages included) to the documentation that you believe supports abandoning this strategy.

Major changes introduced between v10 and v11

  • Shadowing and threshold corrections are now divided by fiber brightness
    • MC files are calibrated by fiber brightness in the following fashion:
      • An x and a y plane of shadowing/threshold corrections are generated for each fiber brightness, for each detector (9 FB values x 2 views = 18 planes per detector)
      • An x and a y plane of attenuation corrections is generated for each fiber brightness, for each detector (9 FB values x 2 views = 18 planes per detector)
      • The muon catcher is treated as its own detector for the purposes of shadowing/threshold and attenuation corrections, giving another 18 planes (bringing the total to 36 for the Near Detector)
      • There is still one atten*.csv file for each fiber brightness in the MC, e.g., from calib_atten_consts.nd.mc.v10.r-r.csv --> calib_atten_consts.nd.mc.v10.r-r.fb{0-8}.csv.
    • Data is calibrated in the same fashion as MC calibrations, with these exceptions:
      • Each cell's attenuation corrections are generated individually. This does not apply to shadowing and threshold corrections, which are MC-driven and applied via mapping to a single x and y plane of corrections, as described above
  • The range for muon-catcher attenuation fits has been corrected
    • This means that muon-catcher attenuation correction functions can be fitted more reliably, without resulting in large numbers of uncalibrated channels due to poor fits.
  • Absolute calibration now derives constants for x and y view independently
    • And then takes the mean as the calibration scale to actually apply

Hacks, deviations, and exceptions to the above Major changes section

  • None expected

Plans in the works for the next calibration cycle

  • Drift calibration corrections may be added
    • This should ensure more uniform FEB response over time (correcting for hardware swaps, hardware deterioration, DSO scans changing the pedestal limits, etc.)