Project

General

Profile

Feature #11750

Interface for noise subtraction service

Added by David Adams over 3 years ago. Updated over 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Start date:
02/17/2016
Due date:
% Done:

0%

Estimated time:
Duration:

Description

We would like to define an interface for a noise subtraction service that would, for example, remove coherent noise from raw data.

A common interface will make it easy to have multiple implementations that are easily exchanged in processing or evaluation code.


Related issues

Blocks dunetpc - Feature #12701: New module and services for raw data preparationClosed05/18/2016

Precedes dunetpc - Feature #12906: Create service to remove coherent noise from ADC dataClosed02/18/201602/18/2016

History

#1 Updated by David Adams over 3 years ago

I suggest the interface receive the data in an expanded, transient format and update it in place. Other tools can be used to unpack the raw data into this format and, if desired, to pack it after such operations are completed.

One possibility is

  TpcData = map<Channel, AdcVector>
  Channel = unsigned int
  AdcVector = vector<short>

A map is used here so one does not have to pass all the data for an event, but could pass only that associated with a TPC plane, readout plane or RCE. The latter is might be the level at which we expect to have common noise.

The interface for a coherent noise subtractor (or ADC code unsticker) could be

  int update(TpcData&) const;

Tools that only handle one channel at a time, would have an interface such as:
  int update(Channel, AdcVector&) const;

Note this does not allow for zero suppression or other missing/bad tick values. If needed, these might be flagged with some special value of short. Longer term (or now?), AdcVector might become a sparse vector.

#2 Updated by David Adams over 3 years ago

The above does not allow for float pedestal subtraction or for the noise removal to have better-than-integer precision. The proposal is amended so that

AdcVector = vector<float>

where float is 32-bit.

#3 Updated by David Adams over 3 years ago

From Brett:

The meta issue is that I want to put the NSS interface into
larsoft-proper while this issue is part of dunetpc. Is targeting
dunetpc just happenstance? Is there a way to transfer the "issue" to
the appropriate larsoft Redmine project?

#4 Updated by David Adams over 3 years ago

Brett:

I would like to retain the dunetpc issue but we can open a separate larsoft issue covering the same point (and referencing this one). As you know, larsoft often moves a slow pace. Dune and uboone can proceed with separate copies of the interface header until larsoft agrees to host the interface. Once larsoft concludes, this resolution of this issue can be to use the larsoft header.

#5 Updated by David Adams over 3 years ago

I had some discussions with Brett re the interface. There are other actions that might share the interface or at least make use of the same data types. The include stuck bit mitigation, conversion from ADC count to ADC float, pedestal addition, deconvolution (in time and/or channel) and ROI building.

In addition to updating the AdcVector, some of these would produce additional information that might be of interest to downstream data consumers. E.g. to mark which channels are underflow/overflow, to identify noisy or unusable ranges within an AdcVector or to flag bins where the original ADC count has been replaced with an interpolation from neighbors.

My current thinking is that it would be best to keep this data with the AdcVector, e.g. to have an AdcData class that holds an AdcVector plus this other information. The types that appear in our tool/service interfaces are then

 
  Channel = unsigned int
  AdcVector = vector<float>
  AdcData = AdcVector + more
  TpcData = map<Channel, AdcData>

The more in AdcData might be fixed fields:


struct AdcData {
  AdcVector data;
  BoolVector overflows;
  BoolVector underflows;
  BoolVector bad;
  BoolVector replaced;
};


implying the class grows as new requirements are recognized.

Or it might just hold named entries with names assigned when the information is added:


BoolVector = vector<bool>
FloatVector = vector<float>
struct AdcData {
  AdcVector data;
  map<string,BoolVector> boolVectors;
  map<string,FloatVector> floatVectors;
};

#6 Updated by David Adams over 3 years ago

I see that DuneInterface/AdcTypes.h (in dunetpc) has this:


typedef double AdcSignal;
typedef std::vector<AdcSignal> AdcSignalVector;
typedef std::vector<AdcSignalVector> AdcSignalVectorVector;

use in the DetSim interfaces. I will try changing double-->float so we can use the same for DetSim and DataPrep.

#7 Updated by David Adams over 3 years ago

  • Blocks Feature #12701: New module and services for raw data preparation added

#8 Updated by David Adams over 3 years ago

  • Precedes Feature #12906: Create service to remove coherent noise from ADC data added

#9 Updated by David Adams over 3 years ago

  • Status changed from Assigned to Closed

An interface for noise removal is now in dunetpc: dunetpc/dune/DuneInterface/AdcNoiseRemovalService.h.

I close this ticket.



Also available in: Atom PDF