Matthew, Michael G., Jeny, Paola, Chris, Joseph, Susan, Nick, Dominick, Bruno, Ruth, Gavin, Ryan, Craig
- We're tagging twice this week, once before the new version of art and once afterwards. In future we will try to follow this weeks model of announcing tags to a wide audience in advance.
- The production tests broke over the weekend due to jobsub depreciation. Now up and running with jobsub_client. The tests now include the merged reco/pid step - something we should use for the next large production of files. Matthew TODO: setup FTS to pick these up.
- Calibration files announced to the collaboration.
- First altered hadronic system files announced to collaboration.
- SAM for users ready for beta-testing - please use it and feedback your impressions to the team.
Are we ready to attempt rapid turn around v6?¶
There have been several recent additions to the simulation that may necessitate a new version of the ND files being produced. A non-exhaustive list of the new features: new ND geometry; new noise modelling; new genie version (from new art version); new flux files.
It seems that some conclusions on this were reached at the nu_e workshop and those conclusions should influence our choice here. Production will hold off pending the output of those discussions.
The latest versions of the flux files (v7) are around 10 times the size of the previous version. This brought Oasis to a standstill when we tried to copy them over. Nova did not violate any policies by doing this, but it appears policies will have to change. There are a number of open threads addressing this issue, the two key paths are:
- Gavin in discussion with Alexander R. thinks that root compression may not have been turned on. This would lead to an increase in file size of this order. Gavin will follow up with Robert H.
- Craig will follow up with Andrew and the Oasis group to define good usage patterns etc going forwards.
There were also a number of suggestions on possible alternate use methods for these files including:
- Teaching them meta-data and putting them into SAM?
Downtime related issues - it everything ready to go?¶
The following open issues remain:
- Slow samweb queries. At nu_e workshop there was still widespread sam tardiness. Some tardiness is apparently related to complicated draining queries. Gavin thinks we can tidy up these somewhat - although it was noted that simple tidying will reduce the veracity of these queries. Gavin was bugging Robert I. and apparently he couldn’t see any obvious reasons for the slow down. On possible cause is a unix kernel feature. Robert created a ticket with admins to remove this. Everyone should keep bugging him with tardy queries.
- Proxy issues? Satish was working with Ken Herner on this. Matthew to contact Ken about this and get some official statement.
- Lack of priorities are hurting Gavin. Could teach ART SAM wrap about something that works to it’s max time limit.
Craig also noted that there was an FTS issue Robert was working on too, concerning lots of files with no tape location.
Simulation hand-over (Ruth)¶
Ruth is currently working on ND overlays, and FD genie. She has had issues with some bugs that are present in the old release she is using. She ended up copying the latest scripts over to her test release. She then switched to a more modern release S15-02-24 - and is experiencing some issue with opportunistic submission.
Following discussions at the 3pm meeting last week, the plan for simulation is to abandon the idea of working in the old releases for now and instead migrate to a more modern release. However this will be affected by the plans discussed in the rapid-turn around section above.
Ongoing Reco (Satish)¶
All samples done.
Ongoing PID (Bruno, Gavin)¶
Bruno he has processed the post-shutdown PID step for the FD NuMI data stream. All but 16 files are complete and he has passed these off to Chris for LEM.
Ongoing LEM (Chris)¶
Chris has done everything given to him.
Ongoing Mix/CAF (Gavin)¶
Gavin handed off two samples and the baseline. Added the QGCS which are in mixing and CAFing now. Waiting on nodes. The final sample has now been handed off to him.
Reco keep-up (Jeny)¶
New jobsub client isn’t ready to work with the proxy. So she prepared all the scripts to run under the old jobs. She will make datasets definitions and circulate these. It was noted that she has included pre-shutdown runs. Excellent.
Complete. The output datasets have been defined and circulated. She noted that 50 files have failed more than 3 times. All of them except one fail very fast. And one that fails after 40 minutes.
SNOW ticket was updated with failing files. Matthew will check then pass to calibration.
- Empty art event from Chris? Should only need a 1 or two line change in make_sim_fcl. Could also use this as an opportunity to spread events more evenly over the run - would require DQ to provide start and end time to us.
- Ryan to stay make_sim_fcl developer.
- Our reported efficiency is very slow. But is this due to server induced latency? Data handling etc?
- DQ meeting last friday - daq2rawdigit changes were signed off on. The consensus is that it is time to redo FD daq2rawdigit. Matthew to prepare ticket. Susan will flip fcl flag needed to write this DQ information out.