SamWiseMinutes1 23 14¶
Forgot to take attendance: Nate, Gavin, Andrew, Craig, Chris, Adam, Eric, Jan, Mark, and others...
*Fill in the matrix!
*Nate: review output from failed reconstructions jobs.
*Nate: LEMSUM files for Chris B are now a priority.
*Can we take a short cut to get Chris files? Filter + LEMSUM without PID to.
*Gavin: test on 1000 Raw2Root -- move onto the major reprocessing.
*Eric will submit test jobs.
*Gavin: Make sure fcl files in CVMFS are okay
*Adam: fix metadata module. tried to commit a fix for the metadata module – fix broke metadata. Fails during the linking stage.
*Eric: Run reconstruction with properly configured fcl.
*Mengel: Add –f option to JobSub SAM wrapper.
*Jan: update novasoft setup script for database reader password
*Dominick: Add metadata to CAF file
- Resubmitted jobs for MC, last time it had bad metadata (too many colons) * 512 jobs submitted – 512 log files back, but not files (96 successes?)
- two problems, left an important option off (reason for 96 success not producing files)
- failures due to CVMFS failures (7000/10000 fermilab nodes could not see CVMFS)
- Jan reports recent issues due to OASIS server (Andrew mentions that there is an OSG ticket)
* Renaim Cronjob is working
* Resubmitted the 512 and 396 were in SAM.
* Reconstruction jobs submitted on the 396 (still a large failure rate)
>* 52 back this morning
>* SAM monitoring for the project claims 271 seen, 139 busy, 70 finished, 62 errors.
>*Are the failures CVMFS?
- Reconstruction step may be up and running.
- Ran reco on Adams Root dataset. Failed on each event (for known reason, fcl configuration), but ran fine. * Would like feature to send extra files long with the job (Mengel — JobSub has this but can't do it in the current wrapper) * CVMFS novasoft setup needs to point to the new db password.
- Ready to hit the go button on Raw2Root
> (Andrew suggests submitting 1000 jobs to see how it goes) * If this works well move to the cronjob * Already running several thousand jobs for processing.
- Rolled back development – will update once metadata module is fixed. * New DocDB password.