Can't open files produced with outputCommands: drop
The NOvA wiki contains example code to drop branches from art output, by adding a line like:
outputCommands: [ "keep *", "drop sim::Particles_geant_*_GenieGen" ]
to the outputs block in the fcl file.
This is suggested in the context of dropping objects which are incompatible with current software, but I'm trying to use it to produce slimmed down versions of our output files by dropping data irrelevant at later processing steps.
This is actually very effective at reducing filesize. There is one hitch however: I can't run over the files I produce this way.
I get this assertion and stack trace:
nova: /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/IO/Root/RootTree.cc:67: void art::RootTree::addBranch(const art::BranchKey&, const art::BranchDescription&, const std::string&): Assertion `prod.present() == (branch != 0)' failed. Program received signal SIGABRT, Aborted. 0x0000003baa630265 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x0000003baa630265 in raise () from /lib64/libc.so.6 #1 0x0000003baa631d10 in abort () from /lib64/libc.so.6 #2 0x0000003baa6296e6 in __assert_fail () from /lib64/libc.so.6 #3 0x00002aaac30f55d8 in art::RootTree::addBranch (this=0x2f37e28, key=..., prod=..., oldBranchName=...) at /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/IO/Root/RootTree.cc:67 #4 0x00002aaac30c8691 in art::RootInputFile::RootInputFile (this=0x2f37cb0, fileName=..., catalogName=..., processConfiguration=..., logicalFileName=..., filePtr=..., origEventID=..., eventsToSkip=0, whichSubRunsToSkip=..., fcip=..., treeCacheSize=0, treeMaxVirtualSize=-1, processingMode=RunsSubRunsAndEvents, forcedRunOffset=0, whichEventsToProcess=..., noEventSort=false, groupSelectorRules=..., dropMergeable=false, duplicateChecker=..., dropDescendants=true) at /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/IO/Root/RootInputFile.cc:201 #5 0x00002aaac30de516 in art::RootInputFileSequence::initFile (this=0x2f331d0, skipBadFiles=false) at /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/IO/Root/RootInputFileSequence.cc:213 #6 0x00002aaac30dd46a in art::RootInputFileSequence::RootInputFileSequence (this=0x2f331d0, pset=..., catalog=..., primarySequence=true, fcip=..., pMode=RunsSubRunsAndEvents, pReg=..., processConfig=...) at /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/IO/Root/RootInputFileSequence.cc:96 #7 0x00002aaac5056308 in art::RootInput::RootInput (this=0x2f31a10, pset=..., desc=...) at /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/IO/Root/RootInput_source.cc:38 #8 0x00002aaac505732c in make (ps=..., desc=...) at /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/IO/Root/RootInput_source.cc:239 #9 0x00002aaaabfab11c in art::InputSourceFactory::makeInputSource (conf=..., desc=...) at /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/Core/InputSourceFactory.cc:55 #10 0x00002aaaabf67969 in art::makeInput (params=..., processName=..., preg=..., areg=...) at /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/Core/EventProcessor.cc:243 #11 0x00002aaaabf691b4 in art::EventProcessor::EventProcessor (this=0xf7a320, pset=...) at /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/Core/EventProcessor.cc:389 #12 0x00002aaaaadc755d in art::run_art (raw_config=...) at /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/Core/run_art.cc:170 #13 0x00002aaaaadc1c36 in novaapp (argc=4, argv=0x7fffffff8c88) at /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/Core/novaapp.cc:155 #14 0x0000000000400cfc in main (argc=4, argv=0x7fffffff8c88) at /archives/nova/novasoft/art_suite/v0_07_04/source/art/art/Framework/Core/nova.cc:4
I know for sure the job I'm running doesn't need to access the product I removed, and you can see there's no nova code in the trace, so we never requested it.
Is there some separate manifest kept in the file? Should the "drop" output command keep that up-to-date? Is there some way I can perform the surgery I want manually?
This happens with
fastCloning true or false.
#3 Updated by Christopher Green about 9 years ago
- Category set to Metadata
- Status changed from Feedback to Resolved
- Assignee changed from Marc Paterno to Christopher Green
- Target version set to 0.07.14
- % Done changed from 0 to 100
The problem has been identified as a regression introduced into ART 0.07.00 for which there was not a test. The problem has been fixed (and a test introduced to prevent future regression) with ART 0.07.14. Release notes and an official NOvA build are in progress right now.
If you have a data file (please include the
.fcl files used to produce as well as to read, and the data file from which it was produced) please provide the requested information and we can verify beyond all doubt that your issue is indeed resolved by this fix before NOvA goes through the effort of updating its development to compile against the new ART. Incidentally, we have verified that no source-level changes are required to the current NOvA code to build against this new version of ART.
It is currently unclear to me whether the files produced with
DropOnInput will be readable after the fix, or whether these files must be reproduced from the originals once more.
Thanks, and our apologies for the regression.