Matthew, Jonathan, Bruno, Chris, Alex, Jon P, Jeny, Adam, Dominick, Nick, Susan, Craig, Gareth, Kanika, Nate, Satish, Paola

News (Matthew)

  • Reco is frozen after FA14-11-25. Jon P. asked to freeze the DB now, but to leave ChannelInfo unfrozen until we can confirm that ongoing issues are fixed. * Production will henceforth run one meeting a week, on Monday at 11.30 CDT, with the Thursday time slot reserved for topical meetings if needed.

FA Status

  • FD data - This still needs another week or so to LEM. During the discussion it was noted that there are still open questions about whether preselection is being applied correctly.
  • FD beam MC 14 DB - ready to start RHC reco
  • FD cosmic MC - 6M events simulated! Ready for MR-brem. Satish will start reco on these this week.
  • ND data / ND genie - rapid turn around (v4) complete, but lacking post-shutdown events due to missing DQ.
  • ND cosmics - reco validation set produced, but it was decided not to run more events as this is already out of date w.r.t. the rapid turn around.

Nova databases for “dummies” (Jon P.)

Some notes:

  • We need a bad channels manager - DQ job, possible replacement identified. * We also need a person to check run history is complete. This job probably falls between DQ and production. * Jon will add a fcl switch to abort if lacking bad channels (as discussed in his slides). Chris noted that other switches of this form exist, but aren't on by default. They should be. * Jon is working on a tech note, which will appear before the end of the year.

Latest ND rapid turn-around (Dominick)

  • CAFs mostly done, caught up in FTS (due to his fault). * A desire to automate the running of production tests was expressed. Matthew can add this. * This dataset is currently missing modern runs as no DQ, waiting on DQ. * Bug in run nova sam in this FA. Fixed by backport.

Reco keep-up (Jeny)

  • Started this on wednesday, but the jobs stayed in idle until now, then got removed by her this morning. Unsure what the problem is. She will work on this and update us.

MC reco update (Satish)

  • RHC hasn’t started yet - email nate to retire old ones. * ND cosmic running - lots crashed. It was decided not to spend too much energy investigating these as the release is already out of date w.r.t. rapid turn around.

Increased sub-run length (Matthew et al.)

  • Paola's calibration files have been successfully process, is was noted that the running time increased from 17 mins to 30 mins. * Matthew presented some slides, docdb-12442: * All looks good, although some non-trivial scaling was observed in certain modules (notably calhit) - this could be explained by the transition to multi-point rather than the extension of subrun length. The longest process is FD cosmic reco, which now takes ~6 hours.


Susan raised the issue of race condition between raw to root and DQ. This involves RAW files that have children created before the DQ is available. This DQ is then not propagated to the children. A short term solution is to extend Joao's scripts to check for these orphans (found by have dq.isgoodrun of NULL). A longer term solution could be available by introducing a DQ DB and join this with SAM DQ - Adam will follow up with Robert I. about this.