Status of Bad Channel Validation » History » Version 143

« Previous - Version 143/145 (diff) - Next » - Current version
Bryan Ramson, 03/19/2020 11:13 AM

Status of Bad Channel Validation


By definition, a channel is an input/output architecture that allows the flow of information. The NOvA Far detector was designed to have 344 064 readout channels (in contrast with the 20 192 of the Near detector). During detector operation, each channel register hits coming either from particle interactions (signal hits) or from noise.

Given a channel, three states are used in the NOvA software framework (docdb-12771)

  • Cold: log base 10 (HitRate) < 0.5.
  • Hot: log base 10 (HitRate) > 3.5.
  • Good: 0.5 < log base 10 (HitRate) < 3.5.

where the HitRate is just the number of hits that a channel collects in a given time. A bad channel is a readout of detector whose hit rate falls outside the range defined as Good. Bad channel maps are created to monitor the fraction
of bad channels (ratio of the number of bad channels divided by the total number of active analysis channels) versus run for data quality purposes and to be used for real conditions MC. These maps are stored in the bad channel map database.

Due to past issues with writing the maps to the database, there will occasionally be mismatches between what the bad channel map was calculated to be and what is in the database. The bad channel validation recalculates the map and compares it against what is in the database to ensure that if there is a mismatch, it is within tolerable limits (note: it is unclear what "tolerable limits" means). Bad channel validation was run manually over small datasets but was moved to a daily automated cron job for easier processing in November 2016. Section 2 of the data quality technote for Ana2019 gives an overview of the issues related to failures in the automated bad channels cron jobs discovered in February 2019.

Validation Status

Red: not validated with autovalidation off
Orange: in progress
Blue: validated and awaiting tag
Green: validated and tagged

The Far Detector

FarDet Data
Dataset name Run range dB tag release Last run in tag Status
N/A 35874-Current ?? ?? AutoValidating
N/A 33542-35873 v23.0 35873 Validated
N/A 32398-33541 v22.0 33541 Validated
N/A 30266-32397 v22.0 ?? AutoValidated
N/A 29428-30266 v21.0 30266 AutoValidated
N/A 28037-29428 v21.0 ?? AutoValidated
N/A 24548-28036 v21.0 ?? AutoValidated
jasq-fd-artdaq-24409-24548 24409-24548 v15.0 24548 Validated*
jasq-fd-artdaq-24039-24408 24039-24408 v15.0 "" Validated*
jasq-fd-artdaq-23978-24038 23978-24038 v15.0 "" Validated
jasq-fd-artdaq-goodruns-23671-23977 23671-23977 v15.0 "" Validated
jasq-fd-artdaq-goodruns-23177-23671 23177-23671 v15.0 "" Validated
jasq-fd-artdaq-22901-23176 22901-23176 v14.0 23176 Validated
jasq-fd-artdaq-22839-22900 22839-22900 v13.0 22900 Validated
jasq-fd-artdaq-goodruns-22717-22838 22717-22838 "" "" Validated
jasq-fd-artdaq-goodruns-22508-22717 22508-22717 v12.0 22717 Validated
jasq-fd-artdaq-goodruns-22385-22507 22385-22507 v11.0 22507 Validated
jasq-fd-artdaq-goodruns-22168-22385 22168-22384 v11.0 "" Validated
jasq-fd-artdaq-goodruns-22088-22167 22088-22167 v11.0 "" Validated
jasq-fd-artdaq-goodruns-22020-22087 22020-22087 v11.0 "" Validated
jasq-fd-artdaq-goodruns-21938-22019 21938-22019 v11.0 "" Validated
jasq-fd-artdaq-goodruns-21882-21937 21882-21937 v10.0 21937 Validated
jasq-fd-artdaq-goodruns-21231-21881 21231-21881 v10.0 "" Validated
jasq-endEpoch3b-fd-artdaq-goodruns-21220-21230 21220-21230 v10.0 "" Validated
jasq-gain150Beam-fd-artdaq-goodruns-20923-21220 20923-21220 v10.0 "" Validated
jasq-gain150NoBeam-fd-artdaq-goodruns-20753-20923 20753-20923 v10.0 "" Validated
jasq-shutdown-fd-artdaq-goodruns-19959-20753 19959-20753 v10.0 "" Validated
jasq-hornscans-fd-artdaq-goodruns-19747-19959 19747-19959 v10.0 "" Validated
jasq-prod_artdaq_fd_numi_fa_goodruns-S15-03-11 12942-17078 v10.0 "" Validated
  • = Still mismatches of the order of <= 1 %. Working on that.
  • = OnMon PixelDCMRate plots are empty, making mismatch around 98% for a couple of subruns.

The Near Detector

NearDet Data
Dataset name Run range dB tag release Last run in tag Status
N/A 13575-Current ?? ?? AutoValidating
N/A 13283-13574 v23.0 13574 AutoValidated
N/A 12795-13282 v22.0 13282 AutoValidated
N/A 11917-12794 v21.0 12794 AutoValidated
jasq-nd-artdaq-11668-11916 11668-11916 v15.0 11916 AutoValidated
jasq-nd-artdaq-goodruns-11621-11668 11621-11668 v15.0 "" Validated
jasq-nd-artdaq-goodruns-11574-11620 11574-11620 v14.0 11620 Validated
jasq-nd-artdaq-11537-11573 11537-11573 v13.0 11573 Validated
jasq-nd-artdaq-goodruns-11504-11536 11504-11536 v13.0 "" Validated
jasq-nd-artdaq-goodruns-11470-11503 11470-11503 v12.0 11503 Validated
jasq-nd-artdaq-goodruns-11451-11470 11451-11470 v11.0 11469 Validated
jasq-nd-artdaq-goodruns-11416-11450 11416-11450 v11.0 "" Validated
jasq-nd-artdaq-goodruns-11384-11415 11384-11415 v11.0 "" Validated
jasq-nd-artdaq-goodruns-11369-11383 11369-11383 v11.0 "" Validated
jasq-nd-artdaq-goodruns-11302-11368 11302-11368 v10.0 11368 Validated
jasq-nd-artdaq-goodruns-11016-11301 11016-11301 v10.0 "" Validated
jasq-test-nd-artdaq-goodruns 10377-11015 v10.0 "" Validated

Note 1: The mismatches of the ~1% level go away when i) you rebuild novasoft ii) try a tag release instead of development.

Note 2: If file has daq2rawdigit.base_release OLD and NEW, it must match with newest version in bad channel database (?)


For quick check purposes, validation can be performed interactively in any nova machine as

% setup_proxy 
% source ChannelInfo/OnMonAna/ 


Some documents for explanations and further details

Bad Channels validation technote

Bad Channel validation results for Far and Near detectors

Bad Channel systematics technote for 1st Analysis

Epoch name conventions, linked here.


From docdb-36531 --
  • Validate older periods
    • 7a-c used tag v18.0. We need to validate the new tag before the 2020 analysis. Check specific subruns for selected events to investigate channels like the one shown in the EVD on the prior slide.
    • There are chunks of missing runs from older periods that were previously validated. Follow up on this to see if these need validation rerun.
  • ND validation
    • all of periods 7 and 8 need validation
    • check older periods for chunks of missing runs
  • Understand where differences in validation come from slight discrepancy between database and OnMon in validation jobs. It’s smaller than it used to be, but should be better understood (and fixed if possible).
  • Check manual bad channel lists: Did these make it into the database for the correct run ranges? Is the current list correct?
    • BadChanList.fcl previously had 7 pairs of swapped channels in the KnownBadChannels list. These channels were removed when they were fixed. Will these channels be properly masked off in older runs if we re-process old files?
    • Current list contains 6 FD and 4 ND channels. First pair of FD channels was fixed in 2015 (AR ECL-5547). Should any of the others still be on the list? Is there a way to go back and properly mask/unmask these where appropriate?

Updated: November 20, 2019