Matthew, Jonathan D., Bruno Z., Dominick, Nick, Susan, Joseph, Paola, Anna, Jeny, Nate, Satish, Jon P., Gavin, Michael, Kanika.


  • Lots of requests for new samples and also reasons for possible remakes of samples have come in following the collaboration meeting. Our strategy will become clearer over the next few days. I feel that it is important to continue to allow high level studies to continue while also working on low level changes
  • ND rock generation stage is fixed thanks to Robert H., although there is a downstream problem in the production tests Matthew will need to fix.

Hadronic sample variation (Nate, Satish)


Generation went fine, some issues occurred with datasets and metadata (just bugs) which Nate has fixed. One of these issues was caused by the fact that there is now a geantonly data tier, so geantonly no longer need to be specified in the special field. The samples are taking about 12 hours each:

  • FTF_BIC is essentially finished
  • FTS_PBERT is next

On Dominick's suggestion, Nate is going to take this opportunity to submit lots of offsite jobs to help us debug why we don’t get as many offsite nodes as we think we should.


A large fraction of the reco jobs crashed with bad validity times, this points to a problem with the times embedded in the events. Nate and Jon will work with Barnalli to establish the cause of the problem here (Edit: this has been tracked down as to do with a timezone issue). Matthew and Jonathan will back port the change to use the weighted times subrun times to work around this and then let Satish know.

It was also noticed that run_nova_sam was not inserting cycle into the file names. Dominick committed a fix to this.

Reco post-shutdown data (Joseph)

Joseph started this yesterday afternoon in the FA14-10-28a release. He reports that it is about halfway done and should finish up overnight tonight. He will communicate this to Bruno and Matthew. He will also work with Dominick to use defman to sort out his definitions. (Dominick will write a defman wiki).

Reco keep up release (Bruno, Jeny)

Bruno wants to run some things by the DQ people and suggests that maybe Monday will be a good time to revisit this. Jeny has also identified a list of issues with this dataset that Bruno and Matthew will look at. She noted that some of the files are taking a while to come through FTS.

Calibration request (Paola)

Calibration want a full scale reprocessing in todays tag, pending validation. There is an outstanding question about whether we need to improve the ND cal-hit time (this is taking longer due to fine-timing optimisation). Paola thinks that we'll be ok to run without this fix, and as Brian thinks it would take order of a week to fix it we've decided to press ahead without it. Matthew will submit the ticket to Paola once the validation is complete.

Production stage merging

Dominick has the pre-LEM part of this ready, needs dropbox fiddling before it is ready to work. Matthew will meet with Nate tomorrow to learn about how to do this.

Bad channels

What are the outstanding issues, what is the path to resolving them, how to work with this:

  • Wrong times in files: this was a problem due to the wrong time being written into the list of good runs provided by DQ and should be resolved now. This affects the files: FD real-conditions GENIE and CRY; ND post-shutdown MC (pre is fine). This is further complicated/mitigated by the fact that bad channels can be configured to chose its validity time by interpolating based on run number. When this was switched to on the files are unaffected. Not true for all FA14-10-28 reco. Rapid turn around got lucky and avoided this. The complication with this “fix” is that files using it were affected by a bug which caused a problem in the dropped DCM logic. This affects releases between Jan 8 and Jan 24th. For details see Nick R. docdb-12689
    • To fix this we either need to re-run reco+ in an old release with UseSubrunForMCTimes: true, or a brand new release…
    • Alternatively we can re-sim the files with the correct time in the file - best, but will take longer
  • Webservice bug reported by Nick, to quote his email “there are instances where a significant fraction (~25%) of the detector has been masked, even though the code that generates the bad channels doesn't indicate any such large scale masking of channels. Jon suspected there may still be issues with db web server, but I'm not sure what the time scale of fixing this would be” - sounds like it needs expert intervention.

Bottom line: bad channels is still not working correctly. Need to fix this, re-generate MC (FD real-condition GENIE and CRY & ND post-shutdown MC), then re-run from scratch. The plan for this will come together over the next couple of days.

Spill level DQ flag

Following some discussion at the collaboration meeting Andrew N. added a check on the number of reporting DCMs at the daq2rawdigits step. Once this has been validated we need to re-raw2root all data… This would then of course require a remake of everything.

ND geometry changes

The ND geometry has been surveyed and is different from what we’ve used to date. Tian has the file and is adding the finishing touches to this. A minimal study is then to re-generate an ND sample with this in and compare to the baseline.

We should also establish whether this should go into the daq2rawdigit running. I’d think not as they probably need some time to validate the new geometry.

Extra samples

I need to poll the various groups for a list of samples they’d like. However I think there is lots up in the air now, so this might have to be put off. email nova_converners


  • Matthew to prepare production epochs.
  • Metadata to change - if sim doesn’t answer shall we just do ours? The consensus is yes we should.
  • Dominick: moved run_nova_sam from metadata to grid utils, run nova sam is in the novasoft bin, and the order of setup is not deterministic, so need to remove one. Dominick will do that. Manually delete 01-21, 02-04