Paul, Bruno, Joseph, Craig, Susan, Chris, David, Alex, Gavin, Enrique, Dominick
There is a Production workshop coming up. It looks likely to be on August 24-25. If you haven't already, please fill out the doodle poll at:
SW Tags (David/Jonathan/Gavin) -- should discuss development build failures¶
David believes he has tracked down the build failure to an issue with inconsistent versions of libwda. However, Gavin thinks this is a symptom of problems related to the change in ART versions. His solution is to roll back ART and related packages to those used in the June 30 tag. We will leave things in this state for about a week, and then move forward.
There was a tag scheduled for tomorrow; but David will hold off until the compilation issue is resolved. This will delay resumption of reco-keepup, but that was deemed acceptable.
Dataset consistency checks (all):¶
Chris raised the issue of verifying that we have a CAF for every PID. And similar for unblinded and decaf files. It's also necessary to validated that the dq flag is correctly transmitted to all files.
Satish has a script to do the checks, and will run them. Update The script only checks for PID/CAF consistency, and needs to be generalized. Satish will do this. In the past, Chris has identified problems
with incorrectly set dq flags for decafs -- in particular for FD NuMI.
FD Topup Stragglers (Bruno)¶
We are still missing two files in the FD topup dataset. Bruno has identified these files and notes that the locations for these files is bad. Chris suspects that this is a symptom of a deeper issue, probably bad (0 event) lempart failures. Update Bruno has sent around one of these files, and Chris has verified that this is indeed the case. These files probably need to be retired and regenerated.
decaf Stragglers (Chris)¶
The FD topup decafs are still missing 28 files + the two topup stragglers mentioned above. Draining has not worked. Chris has tracked this down to FTS04 not functioning properly. FTS restarts appear to have helped, but several attempts were required. He believes that the 28 files have been successfully processed, and is waiting on the FTS to function correctly for the files to show up. It was suggested that if the issue persists, Chris files a ticket.
Production Files on BlueArc vs dCache (all)¶
It was concluded that Alex's suggestion to move files out of their declared path would also require updating the file locations in SAM. Otherwise when SAM attempts to deliver a file, problems would arise when it checks a no longer valid location. It should be possible to generate an FTS rule to handle the files that still need to be copied to BlueArc. There was some discussion about whether this would cause lock-problems or not, and it was concluded that it would not. Satish will contact Robert I to get instructions.
ND CRY generation (Ruth)¶
Alt Intensity Generation (Paul)¶
Jobs are running. Low Intensity is nearly complete (200 files to go). High intensity jobs still have about 2k jobs to go. Paul is waiting for jobs to finish before submitting draining jobs. These jobs still produce files w/ the wrong metadata, and Paul will send Satish instructions on how to fix the issue.
RHC FD MC reco (Enrique)¶
Enrique has been trying to run jobs since Friday. However his jobs have been failing with the missing diblock mask error. The workaround for this should have been backported and published, so this is something of a mystery. Update David has verified that publication of the relevant releases was ok. Kirk's help is now being enlisted.