Chris, Kirk, Matthew, Gavin, Kanika, Satish, Nate, Craig, Adam, Eric, Michael, Dominick, Paula, Jeny, Pavan.


ND raw-to-root keep-up update (Adam)

This is going well with files showing up in SAM. However there is a higher than expected failure rate which seems to be related to data-logger problems whereby files were getting written to the all/wrong trigger stream. It was argued that a good solution to problems of this form might be to run a script which fixes the files meta-data rather than instead having a very messy way of handling files with wrong meta-data.

Action Item: Adam will investigate the pathologies of the data-logger problem and suggest a solution in collaboration with Andrew & Peter.

FD/ND keep-up handover & monitoring (Adam, Jeny, Paula)

Action Item: Adam, Jeny & Paula will meet today (Monday) to work on this.

MC generation handover (Nate, Eric, Jeny, Paula)

Eric is in the process of writing up and will be moving on on the timescale of 3-4 months. This means we need to hand over his work. Eric, Jeny, and Paula met last week and went through Eric’s scripts. However Jeny & Paula's current priorities are rightly getting keep up, up and running. Whether they'll be able to pick this up for the first analysis depends on how fast their other commitments progress.

Action Item: Matthew/all decide on the correct segmentation of CD/experiment when it comes to MC generation.
Action Item: Matthew & Eric will meet so Matthew can get an overview of Eric’s tasks/scripts.
Action Item: Nate will meet with Jeny & Paula to facilitate knowledge exchange on this subject.

FD genie/cry overlays (Nate)

The simulation stage of these jobs is complete and has been circulated. The conclusion is that they look good.

Action Item: Nate will walk these through the remaining stages of the production chain.

FD reco update (Satish)

This is the baseline MC from Neutrino 2014 being updated to make it kosher. This is going well, most of them are through the PID stage. Some issues were observed with FTS directories are not working (LEM files missing), possibly common with something Gavin has seen for the ND CAF files. He is investigating.

Action Item: Satish will complete the processing on these through to CAF.
Action Item: Gavin will investigate the FTS issues.

PC hits with timing calibration files (Evan, Satish, Gavin, Kanika)

In order to include Evan's timing calibration the pchit/stopper light-reco production tier has to be expanded to produce a third output with a data tier of "timecal". Satish merged the required FHiCL paths into the existing FHiCL. Problems were observed during testing due to long run times and memory issues. Probably related to calibration. This begs the question, why are we calibrating hits in the files used to compute our calibrations? The answer is that the stopper selection is cutting on calibrated energy - which creates a circular dependency and can lead to inconsistencies.

Action Item: Matthew will communicate with Marco & the calibration group to push to get this removed.
Action Item: Satish will investigate the right mix of calibration options to reduced his failure rate.
Action Item: All check for consistency within existing & future spotter datasets.

NuMI processing with masking (Dominick)

Dominick is working on CAF failure modes, memory spikes (in collaboration with Igor (CD) & Chris) as well as NuMI processing with/without blinding. This NuMI processing with masking will serve as a dress rehearsal for data processing - including the production of two sets of CAFs, one blinded, one unblinded which we haven't tested on a full scale before. This is dependent on the diblock masking Joao & Kirk are working on which needs to go into the database, metadata etc.

Action Item: Dominick & Chris, (try to) resolve memory spike issues.
Action Item: Dominick exercise the machinery for making the two sets of CAFs & circulate the resultant files to collaborators who could look at these for sideband estimates.
Action Item: Dominick record metrics for the double CAF tier.
Action Item: Matthew incorporate this stage into the testing framework.

Metadata in the new ART version, need for a new service (Adam, Gavin)

The new ART version automatically fills in a lot of the metadata fields for us. However it seems to be doing this "wrong" and also it seems to be overwriting any metadata we add. Adam worked around the providence information with a hack.

Action Item: Adam will commit his work-arounds.
Action Item: Adam & Gavin will collect the problems this update has caused and feed this information back to the Artists with the aim of getting them to fix it within ART rather than us having to proliferate our hacks.

Novapro accounts for production submission (Andrew)

Andrew has setup a novapro account that in future all official production should be done under.

Action Item: Matthew will follow this up with Andrew, Jeny & Paula.

Branching & nightly merging update (Kanika)

There is a cron job setup to do the automatic merging which seems to work.

Action Item: Dominick will work within this branch to test the dual CAF production.
Action Item: Kanika will release the information on this branch to offline when she is happy that it works.
Action Item: Chris gets too many emails.

Beta SAM metrics web site (Michael)

Michael Ghieth has written a very slick beta-production dashboard

Action Item: All, take a look at this and provide Michael with some feedback.


  • Gavin & Kanika have been working on generalised setup scripts.
  • Pavan has (or will shortly have) cleared ~50 TB of space!