General status:

  • Computing issues (Alex)
    • IFDH
      • the server-side update seems to have only affected FA14-10-03x.a.
      • Gavin thinks he has worked out a fix. Will put in touch with Paul S and Paul R
    • Database
      • Some new strange database errors have appeared. Frequent, but not consistent
      • Run with export NOVADBVERB=100 to help diagnose the errors
  • Releases (Paul S)
    • Look into externals setup scripts for nearline
    • Needs to upgrade fifeutils

Processing requests:

  1. Extra FA ND MC (
    • Give Paul Gavin’s fix, run ASAP for Xuebing’s analysis
  2. Topup ND MC status (
    • Also waiting on ifdhc fix
  3. FA respin (
    • Dropping LEMPRESEL (Joe)
      • needs an example problem file
      • Looks like reco may need to be rerun because of modules checking the filter list
      • Order may matter — need to check today
  4. Oct 15 samples (
    • Data Calib Samples (OPOS)
      • Jobs submitted for ND data and FD data
      • ND cosmics almost done (86 spending)
      • ND DD activity jobs failing. No real error message, just a error code 245.
      • Works for some files and not for other files.
      • Planning to see if a retry of a file will succeed later
      • FD cosmic is being processed now
      • All submitted offsite, so may be offsite instability
      • Offsite jobs taking longer than 24 hours to start
      • Bug report: Job tries to run and doesn’t find the samweb project, and doesn’t return an error code (art sam web wrapper somewhere)
    • CRY samples (Paul R, followed by OPOS)
      • Move to new release S15-10-10 release to pick up simulation updates
    • ND genie samples
      • Move to new release S15-10-10 release to pick up simulation updates
  5. raw2root keepup
    • Running smoothly except for 2 files with metadata problems
    • Can just fix the json file or remove and regenerate metadata file
    • Satish suggest a fix tomorrow
    • Bug report: script finished as successful even if it doesn’t not fetch the file
    • Satish update ticket tomorrow
  6. calibration keepup
    • Having trouble because the jobs are taking much longer than 24 hours to start (happening a lot) (May have fixed itself)
    • Ask Robert I.: can we extend the lifetime of samweb projects?
  7. raw2root backprocessing
    • Complete

Ongoing projects:

  • DB code forced crashes (Sijith)
    • docdb-14127
    • Talk to OnMon about crashes in their code - talk to Jon Paley
  • Offsite status report (Enrique)
    • docdb-14039
    • Lots of trouble last week, but this week things are improving
    • Working on getting multi-cpu slots to get a memory increase
  • sam4users update (Joe)
    • On hold due to LEMPRESEL work
  • Adding decaf into Mix/CAF steps (Bing)
    • Missing due to a class, no update