SamWiseMinutes4 17 14¶
Nate, Andrew, Michael, Eric, Kanika, Jan, Ryan, Chris, Dominik, Satish, Tom,
Adam – traveling today
Gavin –also traveing today
Nate work get ND overlay running dedicated offsite resources
Eric sort out the monitoring,
Game plan from last time:
— Run some more MC to exercise GM01 and GM03 —
— FD GENIE is most important - (ND GENIE if FD is finished) — just need overlay specific metadata.
——> FD GENIE mostly done.
— ND overlay is low priority – testing overlay is higher priority. (need online tag for multipoint)
——> Some testing – still some issues. Works as long as you run at fermilab.
——> Offsite may not be ideal with the rock files for overlay? 2 hours, input about 2 GB.
——> ND production size?
——> SMU, OSU, and Harvard have dedicated storage. Big disk array – can register location with SAM, or just pick it up from a directory structure.
- Need calibration and tag to start reconstruction: would be great to run some jobs by 3pm meeting tomorrow (challenging goal)
——> Reco has started…
- data Raw2root jobs if available – wait to start keep-up processing after some data has been reconstructed.
- Dominick will look into CAF/SAM needs
——> Email sent, but focused on RECO…
——> Adam is looking at this.
- Chris and Matthew will follow up with Jon Paley regarding large memory usage for DB access for calibration constants.
——> Passing csv files into the grid job is a work around…
— Met with Pavan
—> identified some files to move.
—> Has some scripts and knows how to monitor and delete.
—> We now have a solution – NOvA archives – backup up to tape, but one directory tree.
—> Easy to reclaim the tape. Claimed 5 TB of space. 17 min for 100 files.
— Waiting on the new ARTDAQ compiled against new ART version…
Nate: testing overlay:
— bunch of FD GENIE — spreadsheet not final, but not ready 4800 entries for 2800 files…
— 95/2800 needed recovers
— ND cosmics worked on AMAZON for pay cluster.
—> metadata is complaining about this metadata.
— Most ran at Fermilab… OASIS downtime…
20,000 Waiting for Tape?
— Michael will follow up with Robert I…
— Tuesday started running things – lots of database format
— Now using CSV files (60 MB total) —> Will grow to 200MB by full detector…
— Running great! Still hitting database for bad channels.
(problems isolated to calibrations table – points table, not const table)
— factors of 10 from 'bad channels' to 'const 'and again to 'points' tables.
— Jobs running great – 20, 100, 200, 500, 1000, ???
—> Need permission to scale up to more than 1,000 — will coordinate with GAVIN.
— Ryan: How much cosmic reco have your processed?
—> Not many cosmics yet. Goal: match MC statistics. Announcement tomorrow with CAFs?
—> Cosmic veto after slice/cosmic track. Heavier reco after veto.
—> NUMI files without veto – 40 minutes per file.
— Does the CAF have the variables that the cosmic veto uses…