Joseph, Satish, Paul, Alex, Nitin, Gavin, Dominick, Chris
This is a final reminder that there will be a production workshop Aug 24-25.
As of Aug 11, novasoft has been removed from OASIS. This does not seem to have caused any issues. Update Some errors seen by Paola may shed doubt on this.
The common UPS products provided by SCD will be removed from OASIS on August 25. We need to ensure that are setup scripts continue to point to the right place. Gavin will follow up to verify for our setup scripts. Satish will check on NovaGridUtils
Computing Issues (all)¶
Late last week nova grid jobs were exhibiting a very low efficiency. This was attributed to several users either reading from or writing to Blue-Arc. Killing the relevant jobs helped, but the problem recurred on Saturday. The recommendation from SCD was that when writing to BlueArc, users should specify the IFDH_STAGE_VIA variable to force ifdh to stage the copied back output file on a local disk or dCache before copying them to the target destination. However, no-one on nova seems to know the appropriate value for this variable. Alex will investigate to find out the answer.
MR Brem Production Integration (Nitin):¶
As implemented, integrating MR Brem into production requires addition of two FCL files, not yet committed to SVN. One creates modified configurations of the standard production packages, and the other defines the job. Chris suggests that settings in the first FCL should be distributed amongst the various module FCLs to ensure that the maintainers of those packages adjust the MR Brem settings as well (not assured, but at least this enhances the likelihood). Nitin will implement this strategy.
David is working to update to new version of art (v1_15_02) later this week.
The generation of additional ND MC is proceeding. At first, he had trouble getting slots, but now he has 700 jobs running. About 2k files have been produced so far. The metadata for these files is incorrect and needs to be fixed before further processing can begin. Paul will send around dataset definitions, and the script to modify the metadata.
Coming down the line, we need to generate ND MC to model the top-up ND data; and ND rock neutrinos with FA14-10-03x.c. Paul has some instructions from Ruth on how to do this, and will coordinate with Gavin to ensure he understands the full process.
Nothing to do at present
Chris is still missing a few files for the top-up ND data. He will follow up with Joseph to ensure these are taken care of. LEMing of RHC fluxswap and tau is done. The jobs for nonswap RHC is also done, but only half of the files have appeared in SAM. Chris believes this is likely because of old bad files still in the dropbox that need to be removed before the correct files can be copied over. He will investigate to confirm this.
- Bruno has just returned from vacation and will start running jobs for ND top-up data shortly
- Gavin will start processing of RHC MC shortly.
CAF Merging script¶
Chris has script that works for merging CAF files that works. He has some questions about appropriate metadata and filenames to use, given that the files will now span multiple subruns. It was suggested that filenames use "ALL" for the subrun field. It will probably be neccessary to generate a new combined-caf tier to distinguish these files from regular cafs. Another question is whose responsibility it will be to run these jobs. This will depend on exactly what stage of processing this will be done at.
Some larger questions were raised. The goal is primarily to speed up analysis jobs, and analyzers tend to use the decafs only, although Chris pointed out that usage patterns in the numu and nue groups differ, so there may be some value in combining at the caf stage.
Dominick pointed out that the number of records to process for regular cafs may be large enough that there is not much to gain from combining CAFs -- whereas combining decafs may offer a substantial benefit. However it would require doing more than merging all subruns for an individual run.
This would require stabilization of the physics group containment cuts, but it's expected that this should be acceptable given realistic second-analysis timelines.
We will need to discuss this in more detail at the workshop next week.