Attending: Alex, Satish, Susan, Paul Rojas, Paul Sail, Joseph, Paola (OPOS), Enrique
The infrastructure has been in reasonably good shape this past week. There was a jobsub server update that didn't cause any trouble to our jobs. We tagged a new version of NovaGridUtils and now submit_nova_art.py can be used to submit with the --test option to use the testing version of NGU.
Paul Rojas asks for a new version of novaproduction to get the changes he introduced to the metadata in the generation script. This would affect all the ND MC generation, so Satish will do this right after the meeting.
FCL dropbox cleanup¶
There are many errors sitting in the dropbox, mostly coming from naming and/or metadata issues. It's considered safe to remove all this files, so Alex will proceed and do it.
It was totally hammered over the weekend, especially the FCL one, but this is a sing of us producing a lot, not of any issues. For the future, we might consider having multiple FCL FTSs.
Worked nicely in the FD over the weekend, but seems to be very slow in the ND due to more complex ART events. The idea is to maintain a constant pool of 200 jobs running on the ND so we don't need to babysit much on them.
Slow SAM queries: seems to be working fine by now. Alex recommends filling a ticket every time there are any issues with SAM¶
raw2root tag accidentally deleted¶
Due to a lack of disk space, development wasn't being copied correctly into the app area. Paul removed old releases, and accidentally removed one that was being used by raw2root. He's almost done rebuilding it.
For the future, he suggests having a spreadsheet of old releases and their usage case, so we can get rid of very old tags. Susan makes the point that we should special-case those that were used in important analyses (e.g., First Analysis) in case we want to reproduce them in the future. Alex thinks that simply by removing all the SL5 builds, which we no longer support, will give us a few months of buffer in which we can think of a good approach about this. Paul will do that.
prod2reco.a, prod2genie.b and S16-03-04 were tagged last week. Probably a new version of prod2reco will be needed once we know which systematics will be used as part of the calibration systematic studies.
ND numi data¶
In slow progress, due to LEMServer slowness. We will maintain about 200 jobs permanently, so this sample will finish in the background
On hold until we release a new version of novaproduction
Small ND reconstruction sample¶
Enrique saw a few memory issues, so he increased the amount of required memory. Alex asks to not produce too many at the same time so we don't put even more stress on LEMServer.
FD genie FCLs¶
Finished right after the conveners meeting on Friday
FD genie with new noise model¶
First 750 files of each tier made, except for a few taus. Will continue to process this until the whole 6000 files are ready
FD cosmic data memory issues¶
Joseph's been busy and hasn't been able to work on this, but he'll do some tests and figure out if issues are actually solved by dropping Break Point Fitter from the cosmic FCL
FD ideal conditions genie¶
Joseph is draining for third time or so, but it's almost complete
Tapasi is ready to start working on this.
Paul Rojas has been handling lots of different FCL files for many tasks, some of them still waiting for the FTS. Everything is documented in the ELog. Alex asks Paul to concentrate in getting the Amazon web server up to speed. Paul says that he's already run some tests without LEM and using a single file and that they were successful. He's now running further tests using 10 files to check that it still works. Paul has the impression that the time to test the Amazon server in an actual generation is very close.
ND genie fluxswap¶
The generation step is almost done except for about 2000 files.
Neardet was able to run fine given that it uses a different release. However, the FD was hit by the removal of the tag. Since this has been already solved, raw2root can resume tonight.
Release will be updated to S16-03-04 to grab the latest calibration and fixes. As a consequence, we will back process since the 1st of February, as the previous keep-ups did not survive long
Any other issues¶
Alex will report and notify everyone the decision about the ND noise model that should be taken on Tuesday. In the meantime, Enrique will generate a sample 2 times the size of First Analysis using the old model. This would either be the baseline or the systematic sample, but will be needed anyway. Once we know which model will be primary, it will be used for the Genie reweighting as well.
Satish will declare the new version of novaproduction as current. It was previously declared as test.
Production will announce files with new and old noise models once they're ready, so analysers can look at them as soon as possible.