Configure dataprep for Iceberg run 5
Run 5 of Iceberg is underway and I am developing dataprep configurations.
#1 Updated by David Adams about 2 months ago
- Assignee set to David Adams
I did a lot of work on this last week. Changes are in dunetpc.
Dataprep sequences are in dune/DataPrep/fcl/iceberg_dataprep_services.fcl. Those for Iceberg 5 have the names
ib5_dataprep_tools_*. These need tuning and testing.
Last week I added bad channels for the two missing FEMBs and put in a rough calibration for run 5a (14 mV/fC, 1.0 us) based on quick looks at a few of the pulser runs: 0.051 ke/(ADC-tick) for all channels. Tom had a more careful look and using his report of 447 ADC-tick/DAC, I changed the gain to 0.0479 ke/ADC-tick. These values are about twice what we had for protoDUNE which is what I expect for halving the shaping time if the ADC gain is the same. Dave C has indicated we will likely increase the ADC gain.
#2 Updated by David Adams about 2 months ago
The tail/pedestal removal is sometimes failing. It was copied from Iceberg 4 and it appears tails are being removed for induction channels. I disabled that for both Iceberg 4 and 5. Note that the tool is still run to improve the pedestal finding. I also raised the signal finder threshold from 0.4 to 1.0 ke as we have much more sample noise in run 5a than 4b presumably due to the change in shaping. There is also noise from pickup and ADCs evident in other channels.
#5 Updated by David Adams about 2 months ago
I have added a rough calibration for run 5b for data soon to be taken with 14 mV/fC gain, 2 us shaping and 900 mV baseline.
Both iceberg_dataprep_services.fcl and iceberg_dataprep_sequences.fcl are in dunetpc. After more validation, I plan to drop the former.
Commit and push current changes to dunetpc.
#6 Updated by David Adams 18 days ago
I have been busy looking at Iceberg data and updating dunetpc code over the last few weeks. There are now many (30?) identified run periods most due to varying amplifier gain and shaping. These are name 5a, 5b, ..., 5q to start and then Cgs, where g and s are gain and shaping indices 0-3 for (4.7, 7.8, 14, 25) mV/fC and (0.5, 1, 2, 3) us. There are calibrations for all of these and pedestal finders for the Cgs variants. And there are decode_reco_xxx for the 5a-q series.
I plan to add a tool to look up the metadata for a run and then call it at run time so we don't need all the latter. Details to follow.
To better study noise, I want to add some noise metrics to the ROI tree. As a first step the metric tool has been updated to provide the option to write the metrics to the ADC channel data. Now I am modifying the ROI builder to record those metrics in the tree.
#7 Updated by Thomas Junk 18 days ago
I have been maintaining an excel spreadsheet with much of the metadata fields in it. I post it to the iceberg e-log:
There's a script on the online machines that makes a .csv file from what's in /data1/dropbox, and I paste it into an accumulating spreadsheet.
#8 Updated by David Adams 18 days ago
Thanks Tom. I have been running a script by hand on each Iceberg event data file as they appear in dcache and updating my run list:
I use the name of the run configuration file along with a little extra information from the log or Shekhar to assign runs to run groups with fixed values of amplifier gain, shaping and baseline. I then have a separate calibration tool instance for each run period.
What I want is to have single calibration tool that uses the run number to find the appropriate calibration at run time. I plan to do this using the run number to find the gain and shaping. Herb has told me the magic to use ifdh to convert run to file name and then file name to the string metadata blob that has much of the metadata.
Do you also parse this blob to get the metadata fields or do you know how to query for individual fields?
#9 Updated by Thomas Junk 18 days ago
Hi David, The fields in the spreadsheet are filled in with a mixture of outputs from config_dumper -P and sam_metadata_dumper, using files on disk on the online iceberg machines. The script that makes the metadata for SAM reads the same values from the files before they go off to FTS (and then to dCache). It should all be equivalent, though you add value by adding extra information from Shekhar.
#10 Updated by David Adams 14 days ago
There is a new calibration for Iceberg 5 decreasing gains by 2.2% to account for the DAC measurement reported by Shanshan last May. This was discussed in yesterday's Iceberg meeting.
AdcChannelMetericTool has been modified to provide the option to write evaluated metrics into the AdcChannelData metadata map and the fcl configurations modified to enable this by default. The AdcRoiToTree configuration I use for Iceberg studies (dunenoise) has been modified to record fields from the signal finder and the noise metrics so they can be used for study and to make selections for moise measurements as also discussed in the meeting yesterday.
I am about to commit these changes.
#11 Updated by David Adams 2 days ago
The sample RMS (metric samRMS) is useful for identifying busy event-channels that should be excluded from noise evaluation. For Iceberg 5 analysis, this and the noise metrics nsgRMS and nsgRMS50 are include in the ROI tree, and noise vs. time is now evaluated from that tree. To enable this, the tool AdcChannelMetric was modified to add the option to write metrics as AdcChannelData metadata and AdcRoiToTree modified to provide the option to record metadata
AdcChannelDftPlotter has been modified to allow selection based on AdcChannelData metatdata. For the DFT plots of interest, the relevant metatdata is that of the grandparent AdcChannelData and so parent pointers and the option to use these to navigate to parent metatdata/attributes are added to AdcChannelData.