Meeting with NOvA Production Conveners 14-Jan-2013

Executive Summary

NOvA's production conveners are requesting help with adapting their production scripts to meet the needs of this project and to more generally match the needs of long term support of NOvA analysis/production.

The case was made that NOvA's personnel do not have the expertise to produce the required scripts and need assistance from SCD/REX or other experts.

Action Items

  1. Conveners to talk to Alex Sousa's new post doc (Adam) regarding being on project for migration and developement of scripts
  2. Conveners to draft a document outlining the current NOvA production system


A.Norman, A. Lyon, Mat Meuther, Gavin Davis, Nat Mayer


The main issue that was brought forward was a concern that was raise late on the 11th the person originally tagged as a possible resource for modifying the productions scripts isn't available nor expert enough in the Fermi systems (and generalized scripting) to work effectively on this.

One possible alternative was discussed in the form of a new postdoc from U. Cincinnati (Adam Arisano) who is joining NOvA and will be stationed at FNAL. The NOvA conveners were asked to approach his adviser to see if this would be a possible position for him to fill. Nate Mayer will follow up by calling Alex Sousa about it.

The actual amount of work that is involved in migrating the scripts was not clear, mainly because there isn't a comprehensive document of what we currently have in NOvA for the production infrastructure. The conveners (Gavin, Mat and Nate) were charged with producing a document addressing what we are actually using so that we can gauge what is involved. They will work on this today.

Adam and Andrew discussed some of the possible avenues that might be pursued for expertise and consulting within REX. They are going to talk to Margaret Votava about this and see what is possible.

One of the things that was discussed were the different steps that NOvA needs to go through right now. A list was made of the different tasks that need to be done or looked into for the production to work. These were:

  1. Perform a check of the NOvA dependencies (libraries and otherwise) to understand where things are being picked up from.
  2. NOvA production people need to test the SAM/ART interface and verify that it is working correctly and matches what NOvA needs.
  3. An area needs to be identified for the SAM system to cache files to.
    1. This area needs to be tested
    2. ALTERNATIVE: NOvA has files registered with SAM on bluearc. We can just use these references for the short term to avoid the need for explicit cache from tape.
  4. The production scripts need to be "walked"
    1. It's not clear at this point what the actual logical flows are in some of the scripts. People need to walk through them and understand their logic and inner workings
    2. A code review needs to be arranged to act on these.
  5. A CVMFS server (and the associated infrastructure will need to be setup at FNAL) There is already movement on this but Andrew will need to talk to Keith about where this is going and the time scales.
  6. The jobsub suite right now can operate in a number of different modes (and all of them are being used differently which is confusing). This appears to cause problems when trying to work with different parts of the NOvA software.
    1. The ways jobsub is used by NOvA need to be examined
    2. One of the methods jobsub uses (the autogeneration of the script to run) should be deprecated. This is so experiment specific that there is no way it is getting it right without hardcoding things in which make it incompatible with running else where.
  7. The Fermigrid ITB needs to be worked with more and the CVMFS system there checked.

Issues were made for many of these tasks