Task #23726

Dataprep update

Added by David Adams about 1 year ago. Updated about 21 hours ago.

Start date:
Due date:
% Done:


Estimated time:


Dataprep run in production has not kept up with some of the changes I have been using in my studies and I would like update it.

These include:
1. Add the new bad and sticky codes shown at the last couple DRA meetings
2. Switch to the new tail removal
3. Ignore flagged sticky codes in the pedestal finder.

I put these all in one ticket because I would like to change one at a time and verify with the CI testing before proceeding to the next.

adcprp_tpp0z_run000001_evt000001.png (1020 KB) adcprp_tpp0z_run000001_evt000001.png APA 3z display without tail removal David Adams, 10/20/2020 08:28 AM
adcprp_tpp0z_run000001_evt000001.png (1.01 MB) adcprp_tpp0z_run000001_evt000001.png APA 3z with the new tail removal David Adams, 10/20/2020 08:29 AM


#1 Updated by David Adams about 1 year ago

  • Assignee set to David Adams
  • Status changed from New to Work in progress

I start with the bad channels. The 17 bad channels show at the Nov 27 DRA meeting ( are already in dunetpc. Before modifying the prolog channelstatus_pdsp in dune/Protodune/singlephase/fcl/channelstatus_pdsp.fcl, I copied it to pdsp_channel_status_2018 in pdsp_channel_status_2018.fcl in
case we want to go back to the old set.

Christoph reported a change between dunetpc revisions 9825e61b and 3f8748a that is presumably due to the new bad channels. We get 0.38% fewer hits in the data CI test presumably because we filter out bad channels prior to wirecell processing.

#2 Updated by David Adams about 1 year ago

I have just pushed another bad channel update corresponding to my Dec 11 presentation. Here is a log snippet:

%MSG-i SimpleChannelStatusService:  Early 27-Dec-2019 12:06:07 CST JobSetup
Loaded from configuration:
  - 156 bad channels
  - 39 noisy channels
  - largest channel ID: 15359, largest present: 15359

As before, I would expect small changes in the wire containers in our test jobs. Christoph, please confirm that you see this. Thank you.

This is all the bad channels I know now.

#3 Updated by Christoph Alt about 1 year ago

Yes, the datareco_protoDUNESP CI test does report a change in the hit collection size between dunetpc revision d69892ff and 5a96a6ef:

1252: < DecoderandReco | gaushit |  | std::vector<recob::Hit> | 46923

1256: > DecoderandReco | gaushit |  | std::vector<recob::Hit> | 46905

Full log:

#4 Updated by David Adams about 1 year ago

Thanks for the report. That small decrease sounds right.

#5 Updated by David Adams 11 months ago

I am back looking at this. I see two places where we define the standard dataprep tool sequences, tools.


I think the definitions are the same there (same names and same sequences). Tingjun, can you explain the purpose of each of these files?

The duplication is not a good idea because one will have to remember to make changes in both places and users like myself are confused about which is actually include in protodune production reco. I propose to move these definitions to a dedicated file, maybe DataPrep/fcl/prototodune_dataprep_services.fcl that is then included in both of the above files.

Any objections?

#6 Updated by David Rivera 11 months ago

Hi David,


is what we use in the refactored simulations for PDSP.

I agree with your suggested change of creating a dedicated file for dataprep to include in both service configuration files. Thanks for noticing this.

#7 Updated by David Adams 11 months ago

I am working on this now. The service configs are being moved to


I am testing that the dataprep results for one data event are unchanged.

I am also moving the simulation configuration protodune_dataprep_tools_sim. Can someone point me to a simulation file I can use to test reco there?



#8 Updated by Tingjun Yang 11 months ago

Here is a MC file:

The standard MC reco fcl file is:

#9 Updated by David Adams 11 months ago

As expected, the new reco looks identical to the old. Compare the new (reco_dataprep.1) and old (reco_dataprep.0) at

I have pushed the mods to take both data and sim from the new config file.

Christoph, when the next CI tests run, could you let us know if any changes appear? Thank you.

#10 Updated by David Adams 11 months ago

Thanks Tingjun. I assume the CI will cover it but I will also do my own check of the sim data using your file.

#11 Updated by David Adams 11 months ago

I confirm the new sim (reco_dataprep_sim.1 at the same URL as above) is the same as the old (reco_dataprep_sim.0).

#12 Updated by David Adams 11 months ago

Changing gears for a minute, I report that PDSP channel 325 (APA 3 u) was added to the 2018 and 2019 noisy channel lists last week. Its pedestal is mostly stuck about 8 ADC counts from occasional forays to the presumably true pedestal position.

A typical waveform can be seen here:

#13 Updated by Christoph Alt 11 months ago

The CI test doesn't show changes in data product sizes.

#14 Updated by David Adams 11 months ago

Christoph, thanks for confirming the last change had no effect.

I have committed another change: the flagging of sticky codes (not the mitigation) is moved to the start of the dataprep tool sequence. This is expected to have no effect on reco and the plots at
indicate this is the case.

To be very careful, I will wail for confirmation no change is seen in the CI tests before moving on to the next change.

#15 Updated by Tingjun Yang 11 months ago

Unfortunately I made another change to hit finder which changed to reconstruction result. Also larsoft version is changed so CI test does not work.

I have backed out my change. Once Christoph updated the dunetpc dependence on larsoft, we can check if David's change has any impact.

#16 Updated by Christoph Alt 11 months ago

The CI test does report a small change in the hit collection and downstream reco:

1259: < DecoderandReco | gaushit | | std::vector<recob::Hit> | 47091
1260: ---
1261: > DecoderandReco | gaushit | | std::vector<recob::Hit> | 47065

This is after moving to larsoft v08_45_00. I don't see changes in the hit finder from larsoft v08_44_00 to larsoft v08_45_00 or other changes in dunetpc that could explain the change in the hit collection, so this is likely due to your commit. You could back your commit out if you want to confirm this.

#17 Updated by Tingjun Yang 11 months ago

Hi David,

Are those changes expected?

#18 Updated by David Adams 10 months ago

No, I did not expect any changes. I will follow Christoph's suggestion and back the change out. --david

#19 Updated by David Adams 10 months ago

I take it back. The pedestal fitter is configured to ignore sticky codes (I forgot about that) and so small changes from the switch are to be expected. The fitter tries to identify and remove sticky codes before fitting but may not always succeed and cannot remove more than one. I believe all is OK now.

#20 Updated by Christoph Alt 10 months ago

I will update the reference files.

#21 Updated by David Adams 5 months ago

As discussed at 8/26 DRA, we would like to correct calibration to account for new measurement of injection capacitance and for estimated out-of-plane charge sharing. A new tool is added AdcRangeSampleScaler with configuration pdsp_gainCorrection is added to accomplish this. The PDSP data dataprep sequences have been updated to include this correction.

The change is pushed to dunetpc.

I have only tested to verify that the tool is run successfully and will do more testing tonight. We should expect a few more hits and slightly higher signals as the gain has been increased by about 5%. Please report here if any changes (good or bad) are seen.

#22 Updated by David Adams 5 months ago

I confirmed with signal strength plots that the scale change is as expected.

#23 Updated by Christoph Alt 5 months ago

The datareco_protoDUNESP CI test reports a small increase in reconstructed hits from 45762 to 46343. I will update the reference files.

#24 Updated by David Adams 3 months ago

Tingjun is ready to move to the new tail remover:

Ok, Thanks David. [In response to my proposal to move now to the new tool.]

Also according to the ProtoDUNE performance paper:
In each TPC channel, the amplifier and ADC are AC-coupled using a high-pass RC filter with a time constant of approximately τRC =1.1 ms (2200 ticks) for collection-plane channels and 3.3 ms (6600 ticks) for induction-plane channels.

Could you update the tool to use 1.1 ms (2200 ticks)? Also could you apply the tool on all collection wires (both TPC side and cryostat side).


My response to the above:

My long term plan was/is to carry out studies to see if there is are benefits to fine tuning the the RC constant and to applying the correction to induction planes and cryostat-site collection. As is often the case, other studies have preempted this. The change to TPC-side collection was made because the improvement was obvious by eye.

The direct effect of tail removal in the collection planes on an isolated signal is small: a 1-2% shift in area if the pedestal is evaluated in regions unaffected by the pedestal and somewhat less when those are included. The effect is much bigger when a signal of interest falls in the tail of a much larger signal.

Switching to 1.1 ms will decrease the maximal isolated signal shift  by 0.015%.

I will make the changes you requested.

#25 Updated by David Adams 3 months ago

The above changes are in dunetpc. Here is the tail removal configuration:

  pdspTailPedRemovalZKe: {
    DecayTime: 2200
    ExcludeChannelRanges: []
    IncludeChannelRanges: ["apa1z", "apa2z", "apa3z", "apa4z", "apa5z", "apa6z", "apa1c", "apa2c", "apa3c", "apa4c", "apa5c", "apa6c"]
    LogLevel: 1
    MaxTick: 6000
    PedDegree: 0
    PedFreqs: []
    PedTick0: 3000
    SignalFlag: 3
    SignalIterationLimit: 10
    SignalTool: "adcTailSignalFinderKe" 
    tool_type: "ExpTailPedRemover" 

This tool is now used by defaults for both data and simulation. I attach displays showing one plane without and with this tail removal.

#26 Updated by David Adams 3 months ago

Changes are committed to dunetpc.

Etienne or Tingjun, could you confirm that next round of CI tests doesn't show any problems?

Thank you.

#27 Updated by David Adams 3 months ago

I noticed the new tail remover was emitting some warning messages (grep "WARNING:"). These were for bad channels. I modified the messages to show the channel status and added config parameter (NoWarnStatuses) that can be used to suppress the messages for specified status flags. The default config suppresses them for bad channels so we no longer see the messages.

#28 Updated by David Adams 3 months ago

The dataprep service now produces a report at the end of job showing the time used in each of the dataprep tools. E.g. in the current version of my test job:

ToolBasedRawDigitPrepService:dtor: Event count: 1
ToolBasedRawDigitPrepService:dtor:  Call count: 6
ToolBasedRawDigitPrepService:dtor: Time report for 8 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:          pdsp_sticky_codes_ped:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:              pd_adcPedestalFit:   6.02 sec/event
ToolBasedRawDigitPrepService:dtor:           adcSampleCalibration:   0.37 sec/event
ToolBasedRawDigitPrepService:dtor:            pdsp_gainCorrection:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:               pdsp_adcMitigate:   0.38 sec/event
ToolBasedRawDigitPrepService:dtor:            pdsp_timingMitigate:   0.01 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.34 sec/event

#29 Updated by David Adams 3 months ago

All changes are pushed to the head of dunetpc. Please let me know right away if there are any problems. If not, I will close this ticket soon.

#30 Updated by Etienne Chardonnet 3 months ago

Hi David,

Nothing alarming on the CI tests (from 46238 to 46479 hits). I will update the ref file

#31 Updated by David Adams 3 months ago

  • Status changed from Work in progress to Closed

Thank you.

#32 Updated by David Adams 3 months ago

I added the Fcl item CallgrindToolNames to ToolBasedRawDigitPrepService. By default, it is everywhere (in dunetpc), set to []. If tool names are added to this list, valgrind profiling (if run) is toggled on and off when each tool is run.

#33 Updated by David Adams 3 months ago

  • Status changed from Closed to Work in progress

I have made a small optimization in the tail remover: symmetric elements of the matrix are copied instead of calculated. There should be no change in results and the tool should run 10% faster. Pls let me know if you see any changes. I just pushed the mod.

I did check that the unit test gives the same result.

#34 Updated by David Adams about 21 hours ago

  • Status changed from Work in progress to Closed

Presumably this is fine.

Also available in: Atom PDF