Task #23726

Dataprep update

Added by David Adams about 2 months ago. Updated 25 days ago.

Work in progress
Start date:
Due date:
% Done:


Estimated time:


Dataprep run in production has not kept up with some of the changes I have been using in my studies and I would like update it.

These include:
1. Add the new bad and sticky codes shown at the last couple DRA meetings
2. Switch to the new tail removal
3. Ignore flagged sticky codes in the pedestal finder.

I put these all in one ticket because I would like to change one at a time and verify with the CI testing before proceeding to the next.


#1 Updated by David Adams about 2 months ago

  • Assignee set to David Adams
  • Status changed from New to Work in progress

I start with the bad channels. The 17 bad channels show at the Nov 27 DRA meeting ( are already in dunetpc. Before modifying the prolog channelstatus_pdsp in dune/Protodune/singlephase/fcl/channelstatus_pdsp.fcl, I copied it to pdsp_channel_status_2018 in pdsp_channel_status_2018.fcl in
case we want to go back to the old set.

Christoph reported a change between dunetpc revisions 9825e61b and 3f8748a that is presumably due to the new bad channels. We get 0.38% fewer hits in the data CI test presumably because we filter out bad channels prior to wirecell processing.

#2 Updated by David Adams 29 days ago

I have just pushed another bad channel update corresponding to my Dec 11 presentation. Here is a log snippet:

%MSG-i SimpleChannelStatusService:  Early 27-Dec-2019 12:06:07 CST JobSetup
Loaded from configuration:
  - 156 bad channels
  - 39 noisy channels
  - largest channel ID: 15359, largest present: 15359

As before, I would expect small changes in the wire containers in our test jobs. Christoph, please confirm that you see this. Thank you.

This is all the bad channels I know now.

#3 Updated by Christoph Alt 26 days ago

Yes, the datareco_protoDUNESP CI test does report a change in the hit collection size between dunetpc revision d69892ff and 5a96a6ef:

1252: < DecoderandReco | gaushit |  | std::vector<recob::Hit> | 46923

1256: > DecoderandReco | gaushit |  | std::vector<recob::Hit> | 46905

Full log:

#4 Updated by David Adams 25 days ago

Thanks for the report. That small decrease sounds right.

Also available in: Atom PDF