Project

General

Profile

Support #24920

art "readFile" errors affecting MicroBooNE production jobs

Added by Matthew Rosenberg about 2 months ago. Updated about 1 month ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
Start date:
09/07/2020
Due date:
% Done:

0%

Estimated time:
Scope:
Internal
Experiment:
-
SSI Package:
Duration:

Description

Hello art experts,

Recently (for the last week or two) we (from the Fermilab MicroBooNE production team) have been seeing a large number of LArSoft jobs fail with the following art error:

IFCatalogInterface destructor:
%MSG-s ArtException:  PostEndJob 05-Sep-2020 03:28:09 UTC ModuleEndJob
cet::exception caught in art
---- OtherArt BEGIN
  ---- LogicError BEGIN
    Source readFile() did not return a valid FileBlock: FileBlock should be valid or readFile() should throw.
  ---- LogicError END
---- OtherArt END
%MSG
Art has completed and will exit with status 1.

This always seems to affect the first 10 - 30% of submitted jobs, but these jobs often run without issue when re-submitted. This has been impacting a large number of different workflows using a variety of different samples of artroot input files (which have all been used in the past without issue), so I don't think the problem is specific to any particular input artroot file or LArSoft workflow.

We are using art version: v3_01_02
and ifdh_art version: v2_07_03 or v2_07_07 (I have seen jobs using both of these versions fail with this readFile error)

Any insights you could provide would be much appreciated! I have attached a full log file for one of these failed jobs in case that would be helpful.

Thanks!
-Matt

readFile_error_log.txt (45.7 KB) readFile_error_log.txt log file for failed job with art error Matthew Rosenberg, 09/07/2020 03:13 PM

History

#1 Updated by Kyle Knoepfel about 2 months ago

  • Scope set to Internal
  • Description updated (diff)
  • Project changed from mrb to art
  • Experiment - added

#2 Updated by Kyle Knoepfel about 1 month ago

  • Status changed from New to Closed
  • Tracker changed from Bug to Support

Closing at request of author as this issue was unrelated to the framework.



Also available in: Atom PDF