Matthew, Bruno, Nate, Shaokai, Jeny, Alex, Chris, Joseph, Satish, Susan, Dominick, Nick, Gareth, Jonathan.
Upcoming DQ changes for production to be aware of:
- Inclusion of the number of microblocks - it seems like this is still being studied and that we're not ready to start any raw2rooting just now.
Upcoming SIM changes for production to be aware of:
- Sag model, this is still under study, along with it's impact to calibration - we will keep an eye on this.
- Noise model studies converged a while back. We will pick these up with the next iteration.
- As have flux changes (are the g4 files ready yet?)
- Also to watch: thresholds (and some DB stuff?)
Disk space issues (Chris, Susan, Craig)¶
/nova/app filled up and this stopped the FTSes from working. Chris spotted this. It seems to be a results of the recent imposition of quotas, and highlights our need to clean up our FTS logs. I was under the impression that Paola put in place automatic log cleaning? This was recommended by members of the production group over the automatic log cleaning provided with the FTS as we could vary the time delay associated with this easily ourselves. Matthew will investigate.
Chris fixed the problem in the short term by compressing log files. He noted that we seem to have a 200GB quota here, which at the current rate we'll hit again in 2 weeks.
Susan’s monitoring can be taught about quotas to spot things like this in future.
Disk monitoring util (Jonathan)¶
Based on Craig’s suggestion Jonathan wrote a fast df util, which takes seconds to run and checks whether disks are getting full. Currently this runs as cronjob under novapro and parses the output of df. It can email people when limits are hit - only me and Jonathan at the moment. The plan is to teach it to look for spikes in usage and then email the production list with a warning, maybe including some hooks on how to find who is responsible.
Ongoing simulation (Nate)¶
QGSP_BERT - files are processed but not in the dataset as n_pass wasn’t passed through. He will fix this and then declare the files. This is the last production that Nate will do!
Simulation handover (Nate, Ruth)¶
Ruth and Nate are iterating with this. Nate has a write up text which he will put this on the wiki page. Nate is optimistic that he and Ruth can achieve their target of producing the ND new geometry sample by the end of the week.
Ongoing LEM (Chris)¶
The estimates Chris sent round by email were accurate and FTF_BIC and the baseline sample were processed but they haven’t come through the FTS. Note: this was fixed by the end of the meeting by an FTS restart. It was also noted that the defman provided datasets were not picking up the files, this was due to a release mismatch. Chris usually just uses the supplied release in the name.
Ongoing PID (Bruno, Gavin)¶
Bruno is still having issues with his permissions. It was discussed that the issue was that the production VOMS request looks a lot like a standard computing account request so it is vulnerable to mis-handling. The correct ticket is that which says new computing account and uses the RHS pulldown for production. VOMS proxy information triggers different pathways. Want nova pro voms proxy. Art Kramer has been helpful with these requests in the past. Matthew and Bruno will contact Art.
Reco keep-up (Jeny)¶
Jeny created a dataset to Feb 17th, then submitted her keepup on the 20th. 10 errors relating to missing runs in the DB. She will post details on the ticket. Jeny is ready to also process the pre-shutdown data and to start the cron jobs. Various delay scenarios were discussed and it was concluded that a delay of 4 days is likely optimal.
Paola reports that most of calibration keep-up is done, but not yet declared to SAM. Matthew will circulate a summary to production and to calibration.
The fate of FA (all)¶
We're ready to remove this now. Jonathan proposes to remove the branch from svn repo as we can always revive this and we can always use the snapshots.
BPF - a proposal to add (Matthew, Michael)¶
Michael Baird has requested that we add break point fitter to the production jobs. Matthew will add in the changes and monitor the performance via the production tests. He will send the metrics to the group form the test when they’re ready.
- tag: tomorrow morning.