Project

General

Profile

Bug #1804

Can't open files produced with outputCommands: drop

Added by Christopher Backhouse about 9 years ago. Updated about 9 years ago.

Status:
Closed
Priority:
Normal
Category:
Metadata
Target version:
Start date:
09/06/2011
Due date:
% Done:

100%

Estimated time:
Occurs In:
Scope:
Internal
Experiment:
-
SSI Package:
Duration:

Description

The NOvA wiki contains example code to drop branches from art output, by adding a line like:

outputCommands: [ "keep *", "drop sim::Particles_geant_*_GenieGen" ]

to the outputs block in the fcl file.

This is suggested in the context of dropping objects which are incompatible with current software, but I'm trying to use it to produce slimmed down versions of our output files by dropping data irrelevant at later processing steps.
This is actually very effective at reducing filesize. There is one hitch however: I can't run over the files I produce this way.

I get this assertion and stack trace:

nova: /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/IO/Root/RootTree.cc:67: void art::RootTree::addBranch(const art::BranchKey&, const art::BranchDescription&, const std::string&): Assertion `prod.present() == (branch != 0)' failed.
Program received signal SIGABRT, Aborted.
0x0000003baa630265 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x0000003baa630265 in raise () from /lib64/libc.so.6
#1  0x0000003baa631d10 in abort () from /lib64/libc.so.6
#2  0x0000003baa6296e6 in __assert_fail () from /lib64/libc.so.6
#3  0x00002aaac30f55d8 in art::RootTree::addBranch (this=0x2f37e28, key=..., prod=..., oldBranchName=...) at /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/IO/Root/RootTree.cc:67
#4  0x00002aaac30c8691 in art::RootInputFile::RootInputFile (this=0x2f37cb0, fileName=..., catalogName=..., processConfiguration=..., logicalFileName=..., filePtr=..., origEventID=..., eventsToSkip=0, whichSubRunsToSkip=..., fcip=..., 
    treeCacheSize=0, treeMaxVirtualSize=-1, processingMode=RunsSubRunsAndEvents, forcedRunOffset=0, whichEventsToProcess=..., noEventSort=false, groupSelectorRules=..., dropMergeable=false, duplicateChecker=..., dropDescendants=true)
    at /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/IO/Root/RootInputFile.cc:201
#5  0x00002aaac30de516 in art::RootInputFileSequence::initFile (this=0x2f331d0, skipBadFiles=false) at /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/IO/Root/RootInputFileSequence.cc:213
#6  0x00002aaac30dd46a in art::RootInputFileSequence::RootInputFileSequence (this=0x2f331d0, pset=..., catalog=..., primarySequence=true, fcip=..., pMode=RunsSubRunsAndEvents, pReg=..., processConfig=...)
    at /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/IO/Root/RootInputFileSequence.cc:96
#7  0x00002aaac5056308 in art::RootInput::RootInput (this=0x2f31a10, pset=..., desc=...) at /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/IO/Root/RootInput_source.cc:38
#8  0x00002aaac505732c in make (ps=..., desc=...) at /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/IO/Root/RootInput_source.cc:239
#9  0x00002aaaabfab11c in art::InputSourceFactory::makeInputSource (conf=..., desc=...) at /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/Core/InputSourceFactory.cc:55
#10 0x00002aaaabf67969 in art::makeInput (params=..., processName=..., preg=..., areg=...) at /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/Core/EventProcessor.cc:243
#11 0x00002aaaabf691b4 in art::EventProcessor::EventProcessor (this=0xf7a320, pset=...) at /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/Core/EventProcessor.cc:389
#12 0x00002aaaaadc755d in art::run_art (raw_config=...) at /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/Core/run_art.cc:170
#13 0x00002aaaaadc1c36 in novaapp (argc=4, argv=0x7fffffff8c88) at /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/Core/novaapp.cc:155
#14 0x0000000000400cfc in main (argc=4, argv=0x7fffffff8c88) at /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/Core/nova.cc:4

I know for sure the job I'm running doesn't need to access the product I removed, and you can see there's no nova code in the trace, so we never requested it.

Is there some separate manifest kept in the file? Should the "drop" output command keep that up-to-date? Is there some way I can perform the surgery I want manually?

This happens with fastCloning true or false.

trim_file_test.fcl (411 Bytes) trim_file_test.fcl Christopher Green, 09/08/2011 05:13 PM
trim_file_test2.fcl (89 Bytes) trim_file_test2.fcl Christopher Green, 09/08/2011 05:13 PM

Associated revisions

Revision 25e759ab (diff)
Added by Christopher Green about 9 years ago

Fix for issue #1804 (defect noted previously and fixed on master with f3b3895).

Revision 8c895769 (diff)
Added by Christopher Green about 9 years ago

Bump version for release with fix got issue #1804.

Revision 60ccd3bf (diff)
Added by Christopher Green about 9 years ago

Improve test to encompass the actual circumstances of issue #1804.

Revision 1913075b (diff)
Added by Christopher Green about 9 years ago

Improve test to encompass the actual circumstances of issue #1804.

Revision edbc5d51 (diff)
Added by Christopher Green about 9 years ago

Revert "Fix for issue #1804 (defect noted previously and fixed on master with f3b3895)."

This reverts commit 25e759ab9216c299f5c698ded1bfebbe6afb25c8.

Revision 3f2fe16d (diff)
Added by Christopher Green about 9 years ago

Set presence correctly per older versions -- really fixes #1804.

Revision e8da69da (diff)
Added by Christopher Green about 9 years ago

Bump version for release with fix for issue #1757 and real fix for issue #1804.

History

#1 Updated by Marc Paterno about 9 years ago

  • Status changed from New to Assigned
  • Assignee set to Marc Paterno

I am investigating the reported failure.

#2 Updated by Marc Paterno about 9 years ago

  • Status changed from Assigned to Feedback

Hi Christopher,

I am still trying to reproduce the failure you observed. Could you please:

  • let us know exactly what NOvA release you are using, and * post the FHiCL file you used to run this test.

Thanks.

#3 Updated by Christopher Green about 9 years ago

  • Category set to Metadata
  • Status changed from Feedback to Resolved
  • Assignee changed from Marc Paterno to Christopher Green
  • Target version set to 0.07.14
  • % Done changed from 0 to 100

Hi,

The problem has been identified as a regression introduced into ART 0.07.00 for which there was not a test. The problem has been fixed (and a test introduced to prevent future regression) with ART 0.07.14. Release notes and an official NOvA build are in progress right now.

If you have a data file (please include the .fcl files used to produce as well as to read, and the data file from which it was produced) please provide the requested information and we can verify beyond all doubt that your issue is indeed resolved by this fix before NOvA goes through the effort of updating its development to compile against the new ART. Incidentally, we have verified that no source-level changes are required to the current NOvA code to build against this new version of ART.

It is currently unclear to me whether the files produced with DropOnInput will be readable after the fix, or whether these files must be reproduced from the originals once more.

Thanks, and our apologies for the regression.

#4 Updated by Christopher Green about 9 years ago

Upon further examination, the fix, while addressing an ART test failure, did not address your issue specifically. A further fix has been made and tested against your trim_test_file.fcl. This will be released as 0.07.15 along with the fix to issue #1757.

Sorry for the inconvenience,

Chris.

#5 Updated by Christopher Green about 9 years ago

  • Status changed from Resolved to Closed


Also available in: Atom PDF