Matthew, Chris, Gareth, Jonathan D., Gavin, Satish, Jon P., Kanika, Raphael, Susan, Nick, Dominick, Michael, Paola, Michael B. Nate, Craig, Ryan, Pavan.
- The Thursday room for the production meeting is WH3NW not NE. * There was a new tag created yesterday: S14-10-15. This tag will be used for a third round of reco-keep-up. * The calibration group have requested a significant increase in ND cosmic MC statistics from 1M events to 100M events. According to the projected running times this is acceptable and will lead to the processing of this chain through all tiers now taking ~0.15 kCPU weeks (for contrast 10/64 of the FD cosmic data will take ~0.6 kCPU weeks). * Matthew reports that his testing related cron jobs have all stopped working. The consensus was that this could be related to the "aklog" kerberos warning that is appearing on the VMs. He will switch these to run under nova-pro along with the keep-up dataset sample provider as those jobs are still working. * Raphael will add the production meetings to the NOvA google calendar.
SIM production update (Nate)¶
In general simulation submission and running has been going well, however the slow PNFS system is proving to be a significant handicap. Our FTS rates are around a factor of 10 slower than they should be, this is causing all the produced files to become backed up in drop boxes waiting for declaration to SAM and transfer. There is an open ticket but traffic on it has wained.
Action Item: Craig will call Andrew to press this issue. Matthew will follow up on the ticket.
Nate also reported that he found a bug in make_sim_fcl due to MCCheckout keep-up, which was picked up as development work had gone on after he thought that it was frozen. It is fixed now. He is also having issues with priority (he has lots of jobs!), he'll look into prioritising things using methods described by Chris and Ryan.
- The ND genie files could be done by tomorrow afternoon. * Increased ND cosmics statistics will be submitted via our normal methods rather than relying on AWS tests and will be made a priority.
Reco validation discussion (all)¶
Ryan reports that people are looking at files Gavin produced and some progress is being made, although he notes that some peculiarities have been seen:
- The EVD has been crashing in cases where neutrino interactions are in masked di-blocks. Experts are aware and it sounds like a fix has been committed. * The FD light level seems to be low, this seems to be related to Adam's last round of tuning. The good news it that it looks like calibration is still returning a valid solution although there are some extra peculiarities in the percent region. The differences are clear when you look at the difference between small calibration related steps, but final calibration closure itself looks good. It seems as if MC calibration will need to be redone and this will delay reco. Ryan mentioned that we could go ahead with the reco anyway if this is only a few percent effect. * A follow on question is then whether the FD cry files produced are enough for calibration? Ryan notes that the 4M events produced convoluted with the "random runs" detector acceptance likely leads to 3M effective events. * It was unclear whether the masking is being checked?
Generally things are looking good and now we're waiting on updated calibration (which is in turn waiting on new files).
Following SCDs upgrade to a post-gres SAM backend, the FTS version will need to be upgraded. Nate wants to switch to v4_1 rather than v4_0_1 as this includes option to skip checksums which could speed it up. Nate will implement this today. He also mentioned that he spawned a 4th instance, only for FHICL files. Chris would like a priority list in FTS’s in future.
Action Item: Matthew will send emails and get the priority queue feature request in.
Tagging schema discussion (Jonathan)¶
The plan is to start regularly creating tags on a Tuesday every two weeks (automating as much as possible as we go along). In the Monday preceding a tag the production group will decide on an appropriate description of the major changes that go into this tag based on the svn commit history.
Reco keep-up update (Dominick)¶
A new round of keep-up with ND calibration is done. It sounds like a third round is needed to overcome problems Chris identified during his Neutrino hunt, that is problems with the blinding; nu_e cosmic rejection and Kirk's uncontained ana. Dominick will assess the new tag and see if this is appropriate to use for this.
PC list keep-up update (Paola)¶
The jobs for the second round of validation are done, they're just sitting in the drop box waiting for declaration to SAM.
Action Item: Matthew will contact Evan to determine if the minimal set available now are enough for validation.
AWS tests (Paola)¶
These will happen today and tomorrow and the results will be used for SCD to apply for an Amazon grant. We will not rely on this producing usable files for our analysis.
- All disks are at <90% - excellent. * Pavan finished archiving all the CAF files -/nova/ana. looks in good shape. 5TB available space. * Zukai is switched over to PNFS * We should still implement the updates to monitoring scripts that are differential in time (Matthew & Craig)