Craig, Raphael, Andrew, Gavin, Himansu, Robert, Nate, Chris
On Thursday 1st August there were 4100/9000 CAF files.
Chris estimates finishing Tuesday through LEM.
Update: LEM processing finished Tuesday as predicted. Availability to be announced once CAF completes, so Wednesday 7th.
Question about process label and how did it change. Unresolved.
Action Item: Gavin/Nate will look into how the process label changed during production
Last week Nate and Adam did tests committing 1500 fcl files to SAM and submitted a job using dataset as defined by these fcl's. Minor bug - non-unique output filenames. Still a work in progress.
Action Item: Define output name in fcl file generator. Adam to test on return from Snowmass.
Cosmic Overlay concerns¶
Takes 2-4 hours to process 200 spill cosmics file. Overlay with a 3000 spill beam file. 19 hours of reconstruction.
Need to follow up with Lisa on beam metadata progress. Update at next week.
Raphael updated us on CVMFS status.
Problems with permissions. Latest version of cvmfs is very strict and permissions need to be set correctly on the Stratum server.
Lots of work by Raphael to figure this out. Can now setup novasoft anywhere using latest cvmfs.
Also fixed the issue of missing libraries on the FNAL ITB server. Sets up a ups product that carries the "missing" libraries.
Harvard has issues with jobsub due to machines upgrading to SL6 BUT the gatekeeper remains on SL5.
SMU working with the ifdh file transfer. Harvard stuck until gatekeeper is upgraded.
Goal is to start running large-scale tests over the two GRIDs at FNAL and Harvard/SMU/Ohio in order to demonstrate that this works.
Future LEM production¶Long discussion to be taken offline.
Possible solution: Instead of shipping the LEM pid info back to FNAL as a file, we could define a database that contains the LEM info within in it. Then, when the LEM pid object needs to be filled, either at the caf or during an art file analysis the database could just be queried for the LEM value for that specific tag/run/subrun/event/etc. Details still need to be hashed out.
Chris voiced his concerns and has another solution:
- lemsum file is copied back to Fermilab and declared to SAM
- Job submitted over SAM lemsum files runs
- Job queries SAM for a file matching the lemsum metadata, but for a
- Job does the merge, declares the result to SAM
He can imagine several possible variants. Ideally SAM would allow one to express the dependency on two matching input files being available, but from my brief reading this doesn't seem to be a feature. Future smaller focused meeting to discuss this further.
Action Item: Adam and others to provide Chris some feedback and information to better understand limitations of LEM and SAM.
Action Item: Arrange a meeting between key players to discuss this in more detail.
Jan working with Jon to fix Database-related failures in the build.
Action Item: Tian to start PCHits processing for calibration week of 5th August as well as take over raw-to-root processing from Jarek.