Matthew, Joseph, Jeny, Chris, Susan, Nick, Dominick, Alex, Vito, Ruth, Craig, Ryan, Gavin, Satish, Kirk
- Production tests back to normal
- Issue with race conditions between services means that MC with MC run numbers breaks in the last two tags - see email thread.
- Stand up meetings working well propose to keep them running for this week
- Perhaps ask FNAL to push next months maintenance day if needed? Will have to communicate this to them well in advance.
- Send more jobs to Prague FZU.
- Overlay processing is going well, almost at 30%. Sim group are looking at things now. Using all offsite slots.
- ETA at least another week. If needed could submit onsite and get another 2k odd nodes.
- Gavin and others noted that copyback is taking ages - it was infact broken over the weekend - and this can cause jobs to time out. Satish is also seeing new errors relating to this - he has opened new tickets.
- Satish's reco jobs are draining off of Gavin
Other points of note:
- FCL files should be on dCache not blue arc
- nova.standard not set to true for MC
Calibration bug and implications for processing (Chris et al.)¶
- ~50% failure rate seen on FD data and probably the same on ND data - awaiting confirmation.
- Chris estimates 10s to 100s of channels in FD are failing.
- MC in not principle immune, but in practice it seems to be.
- Chris would like to add a debugging tool to check for this sort of thing in calibration. Alex agrees.
The correct way to deal with this bug is to re-run all reco, the wrong way is to patch LEM to ignore this. The decisions was taken to re-start data reco (and later the same decision was taken for MC). We will CAF what we have now to give the analysers something to look at in the short term, but we will not continue making new files, or clean up what we have made.
Reco (Joseph, Satish)¶
- FD Data ETA 6 days, but due to the above we will stop here and restart.
- ND numi still needs cleaning up and draining - won’t bother with the draining for this set.
- MC is going smoothly except for the copyback thing.
- FD genie done, ND not far behind. Think we’re being fast enough that LEM won't be a bottleneck.
CAF (Bruno, Gavin)¶
- FD genie limited CAF is missing some files - actual issue or do we drain? Gavin is cleaning up now at 90+%
- >10% in ND overlay limited CAFs. Dataset will get renamed and then circulated.
- UMN grid is magically working. Susan reports that she should be tuned numue for FD tonight or tomorrow morning. ND done by the end of the week.
- Kirk will be done with cosrej by then end of today.
Getting reco keep-up restarted¶
- Many interested parties are not here, so this item was bumped to an email thread.
- RAW2root Policy for large files/Snews - email DDT.
- Memory usage jump in FD needs to be pinned down.