Matthew, Anna, Paola, Jeny, Quilan Huang, Vito Di Benedetto, Chris, Susan, Nick, Joseph, Alex, Gavin, Satish, Ruth, Ryan.
- Thanks to everyone who covered production in my absence, good to see new tags being made and things progressing. * A new tag was made last week FA14-10-03x which aims to be the tag we use for FA simulation. (Should have been named FA14-10-03a.b - but naming is a minor point) thanks to those who worked hard to pull this together while Jonathan was on holiday. Numerous fixes see: https://cdcvs.fnal.gov/redmine/projects/novaart/wiki/History_of_Tagged_Releases#FA14-10-03x-The-Madness-Continues-Release * Problem identified with calibration fcl. Fixed by Chris and back ported… to S15-02-09 (not FA?) * Bruno reports that he’s been able to make blinded and unblinded CAF files for the post-shutdown FD data. Will circulate definition when the files are done. * Responses to nova-pro role in, will act on those this week. * Up-coming collaboration meeting. Plan to ask for a SAM for users tutorial and updates on flux file providing and the oasis server. * The fermilab computing division had an AWS grant approved - likely very good news for NOvA production.
FA plans (all)¶
- Simulation and reco waiting on light levels tuning to complete. * Once light level tuning is complete the nu_mu energy estimators will be retuned by UMN without production intervention. * Bad channels checking complete. * The hope is that the current calibration will be sufficient. That said a careful check will be done once the above and the simulation are in-hand.
Simulation update (Ruth, Gavin)¶
Some general points:
- The release intended for production, FA14-10-03x, is not on CVMFS yet. Gavin needs it to be there in order to complete his tests. We should note that this release is not complete yet as we’re still short on Adam’s Light level tuning. * Some validation of this release has been completed including test track IDs. * Are there enough rock singles to meet to required repetition rate for the requested sample size? Ruth to start email thread on this. She estimates that the current sample size will give us a repetition rate of between 300 and 500 (not the 200 discussed).
Ruth reported that all the issues that have held up simulation are fixed and in fact these turned out to be a suit of issues:
- No SL6 machines at SMU and then a library not found on their SL5 release (Chris remembers seeing this issue a while ago - it is a missing library at THEIR end, so we need to reach out to SMU - we could also do with a champion for our libraries). * There was an issue with the types of machines at Czech institute (to do with disk space request, default to lower 25Gb). * There was a temporary issue at OSC to do with a ticket. * Fermilab nodes are now called “Fermigrid” * Harvard was down.
But it now all works with the old style OASIS CVMFS flux files.
Oasis migration status (Gavin)¶
Gavin has been working on putting a fix in place with enable us to use flux files in the old release we intend to run production in. This is done via a custom version of art_sam_wrap which handles all of the delivering of files as well as the modification of the fcl to be run to point in the correct location. This work has been done in conjunction with Robert Hatcher and is now working well. The flux files in question are currently sitting on a pnfs location from where they are copied to the grid worker nodes as needed. The copying works onsite and offsite, but currently only with code hosted by the OASIS server (the old one) as new server isn’t mounted offsite.
- Matthew to contact Jonathan and Andrew to see if we can get this remedied.
Gavin notes that if we wanted to do something like this in a generic art_sam_wrap then we'd need a pre-execute hook. This can’t happen in the pre-script as the fcl file needs to have been delivered first. We are not ready to move away from the old server yet so have pushed back on the OSG deadline. A mis-communication with OASIS folks meant they thought we were ready to transition, but we aren’t and Gavin has clarified this with them.
The development interface uses the GENIE helper which is different from the above.
- Is there a hook for users to use the new CVMFS - a new packing of nova-grid utils? * Will have to watch for people trying to run simulation in releases between this tag and development.
Offsite monitoring update (Jeny)¶
See Jeny's docdb: http://nova-docdb.fnal.gov:8080/cgi-bin/ShowDocument?docid=13160
Calibration want to be able to compute drift. It seems we only process post-shutdown FD cosmic data for calibration. There is a request in to include older stuff. However there were open questions remain about the daq2raw digits base release to use. After some iteration Matthew and Paola concluded that the number of overlapping files was very small so we should proceed with the bulk submission as normal and then where files are present in multiple releases, default to the most modern one.
- Matthew needs to meet Anna (Wednesday). New version of jobsub ready.