Attending: Alex, Bruno, Satish, Paola, Felipe, Susan, Vlad, Joseph, Enrique, Cristiana, Alex Radovic, Chris

Computing issues

  • Sam has been sub-performing again. This is partly due to us putting very high load on the system, but it looks like other experiments are doing it too at the same time. As a temporary solution, NOvA has been migrated to it's own server now. There's a chance we find a more clever solution in the future, but the reality is that we are actually the main users. As usual, the advice is to report whenever we observe sam instabilities.
  • Otherwise, the system seems to have been performing fine during the past few days, except for some minor dcache interruptions


  • Paul has been working on a new tag to include GEANT4. Unfortunately, he needs to build using root v6 for that, which it doesn't tie up well with our current version of nutools. Alex asks Paul to try and build novadaq and novaddt with this particular release and use it as a benchmark

h2 Samples

h3 FD data

The jobs are ongoing, but there still aren't definitions, so Joseph will work on them. From staring at the sam station, it looks like there's been reasonable progress, but a non-negligible number of errors too, so Joseph and Cristiana will have a look at them. Draining definitions are now needed too to keep working on these.


About 1/4 already through, with about 500 errors at which Adam will have a look soon. It's now obvious that the FD data put a small enough stress in LEMServer to increase the number of jobs for this sample. The request is to increase it to 300 and keep watching.

FD Cosmics

These will be on hold until the FD data have been processed


Bruno has reprocessed the ND data with the latest release and FD MC

MRE systematics

Alex has made all the artdaq for the Birks systematic, with just a few stragglers. He'll also handle the rest of the systematics, but won't run the full chain until LEMServer is free. They will probably put a lot of stress on LEMServer, so it's easier to us to maintain just one big sample ongoing than trying to synchronise two samples with high load on LEMServer. Therefore, we need to discuss whether to address ND flux swap or MRE systematics first.

raw2root and reco keepup

raw2root had some minor issues on Moday, due to timing outs in sam, but they worked OK after retrying. As a consequence, we're up-to-date with the processing now. We also had minor issues with reco-keepup on Saturday, but they seem to be working fine today.


The quota for novapro is almost full due to a directory in nova offline that's taking all of it. Felipe will send this around to the production list, but it's very likely just logs. Alex will work on redirecting this to the ana area instead. Something else that may help is figuring out why opencv is so big, so Evan and Chris are going to have a look.

There are continuous requests to bring back development to cvmfs. We need to find a good system to not overwrite the files, but this is definitely something we will bring back after the production campaign is over.

Infrastructure improvements

Satish has all the basic scripts for the refurbishing of the production website in place. Basically, there are two scripts at top level, one in charge of downloading the information by doing samweb queries and creating pickle files, and the other just creating webpages. They both check the same configuration files to figure out what they need to be based on, and that's what one would need to change to introduce new definitions. He's been also working on a new system for caching definitions. His plan is to implement a system in which we retry every 72 hours or so, to see whether there's any difference between the cached value and the actual query. Finally, he's interested in rearranging the landing level website to make it more friendly. Alex will try out the new system and give Satish feedback.

Any other business