Project

General

Profile

Trouble Shooting

Recursive #include directive

This type of error message:

Failed to parse the configuration file 'job/t962g4ana.fcl' with exception
---- Recursive #include directive: BEGIN
job/simulationservices.fcl => /grid/fermiapp/lbne/lar/code/larsoft/releases/development/job/simulationservices.fcl
included from line 2 of file "./job/t962g4ana.fcl" 
---- Recursive #include directive: END

is telling you that you have included the indicated .fcl file twice somehow in your job's .fcl. Most likely one of the other included files also includes the offending file. Simply remove the offending file from your job .fcl file. The message also tells you which line of your job .fcl file to fix, in this case line 2.

Malformed #include directive

The following type of error message:

Failed to parse the configuration file 'job/genie_sn.fcl' with exception
---- Malformed #include directive: BEGIN
  #include "job/services.fcl" 
   at line 1 of file ./job/genie_sn.fcl
---- Malformed #include directive: END

can result from an fcl file written with ASCII text being in the DOS format and not the UNIX format. This sometimes happens through the course of an fcl file's transfer. To convert the fcl file to the correct format, run:

dos2unix job/genie_sn.fcl

Service unable to find requested service with compiler type ...

The following type of error message:

%MSG-e BeginJob:  PolkaDots:polkadots@BeginJob 18-Feb-2011 13:36:26 CST  MF-online :0
A cet::exception is going through WorkerT<EDAnalyzer>:

%MSG
%MSG-s ArtException:  PolkaDots:polkadots@BeginJob 18-Feb-2011 13:36:26 CST  MF-online
cet::exception caught in art
---- EventProcessorFailure BEGIN
An exception occurred during current event processing
---- NotFound BEGIN
Service  unable to find requested service with compiler type name 'art::TFileService'.
---- NotFound END
cet::exception caught in EventProcessor and rethrown
---- EventProcessorFailure END
%MSG

is telling you that the framework was unable to locate a service of some type when it was requested. That typically means that you have not specified the service in the services block of your fcl file.

So, in this case the missing service is the art::TFileService and you should check your fcl file to be sure it is defined in the services block. Or check the included fcl files and the user to make sure it was defined in both of those locations.

Undefined symbol errors

These errors typically have messages of the form:

%MSG-s ARTException:  LArRawInputSource:source{*ctor*} 13-Oct-2010 15:45:12 CDT  pre-events
art::Exception caught in mute
---- PluginLibraryLoadError BEGIN
unable to load libLArRawInputSource_module.so because /argoneut/app/users/brebel/artsrt/test/lib/Linux2.6-GCC/libRawDataUtils.so: undefined symbol: _ZN3edm14RawInputSource15getNextItemTypeEvError occurred while creating source LArRawInputSource
---- PluginLibraryLoadError END

%MSG

First Possible Solution

Notice that an undefined symbol was named, _ZN3art14RawInputSource15getNextItemTypeEvError.

This problem is the result missing a library in the link list for building the libLArRawInputSource_module.so. To figure out what the problem is, you need to identify the library that the symbol comes from, ie what the symbol means. To figure out what the symbol is, do

% c++filt symbol

For this example

% c++filt _ZN3art14RawInputSource15getNextItemTypeEvError
art::RawInputSource::getNextItemType()

We see that the symbol is edm::RawInputSource::getNextItemType(), a symbol defined in the edm::RawInputSource. The solution is to put the correct library into the CMakeLists.txt file list of libraries to link against.

Second Possible Solution

Perhaps the undefined symbol is because the symbol is declared in the header file of the object in question but not implemented in the .cxx file. If that is the case, either implement the method in the .cxx file or remove it from the header file.

No dictionary for class

Errors of this kind are produced at run time and typically look like

%MSG-s ARTException:  PostModule 14-Oct-2010 16:54:39 CDT Run: 1 Event: 1
art::Exception caught in mute
---- EventProcessorFailure BEGIN
EventProcessingStopped
---- ScheduleExecutionFailure BEGIN
ProcessingStopped
---- DictionaryNotFound BEGIN
NoMatch TypeID::className: No dictionary for class St6vectorIPN4simb7MCTruthESaIS2_EE
art::Exception going through module NeutrinoAna/neutrinoana run: 1 event: 1
---- DictionaryNotFound END
Exception going through path doit
---- ScheduleExecutionFailure END
an exception occurred during current event processing
art::Exception caught in EventProcessor and rethrown
---- EventProcessorFailure END

%MSG

The error is the result of not linking the proper libraries for your .so file. The solution is to figure out first what the symbol means

$c++filt St6vectorIPN4simb7MCTruthESaIS2_EE
std::vector<simb::MCTruth>

So the symbol is a vector of simb::MCTruth objects, thus you need the library corresponding to the simb namespace, ie libSimulationBase.so.

Now add the library to the list of libraries in the CMakeLists.txt file.

No Reflex data dictionary found

Errors of this form are also produced at run time and look like

%MSG-s ArtException:  EVD:evdisp@Construction 14-Apr-2012 16:04:41 CDT  ModuleConstruction
cet::exception caught in art
---- DictionaryNotFound BEGIN
 No REFLEX data dictionary found for the following classes:

 <insert class name here>

 Most likely each dictionary was never generated,
 but it may be that it was generated in the wrong package.
 Please add (or move) the specification
<class name="whatever"/>
 to the appropriate classes_def.xml file.
 If the class is a template instance, you may need
 to define a dummy variable of this type in classes.h.
 Also, if this class has any transient members,
 you need to specify them in classes_def.xml.
---- DictionaryNotFound END
%MSG

The error message is telling you that you never generated the dictionary for that class indicated by the "<insert class name here>" and is telling you what you need to add to the classes_def.xml file. The possible confusion you might have is which classes_def.xml file is the culprit. To determine that, just use ack with the class name in the top level of your srcs directory:

 
ack class_name $MRB_SOURCE/

and look to see which files refer to the problem class_name.

Failed to parse the configuration file

The FHICL parser will tell you if there are errors in your configuration file. One example is

Failed to parse the configuration file 'job/somejob.fcl' with exception 
---- Can't find key BEGIN
some_parameter (at part "some_parameter")
---- Can't find key END

The message is telling you that the configuration file somejob.fcl is missing a definition of the key some_parameter. The solution is to make sure the key is defined somewhere, possibly in a #include file.

Failed to parse the configuration file on line and character

If the error message is specific to a BEGIN_PROLOG line like

---- Parse error BEGIN
  detected at or near line 1, character 1, of file "someinclude.fcl" 
  included from line 2 of file "somejob.fcl" 
---- Parse error END

then you may have failed to add a colon (:) on a top-level object somewhere in the BEGIN_PROLOG/END_PROLOG block. Look for something like

standard_moduleconfig
{
   module_type: "SomeModule" 
}

Notice the missing colon on standard_moduleconfig.

The error message seems to be insensitive to exactly where.

Can't find key BEGIN

These errors have the form

%MSG-s ArtException:  nova 27-May-2011 10:45:46 CDT JobSetup
cet::exception caught in art
---- Can't find key BEGIN
 some_parameter
---- Can't find key END
%MSG

and are telling you that the configuration file (ie xxx.fcl) is missing a definition of the key some_parameter. The solution is to make sure the key is defined somewhere, possibly in a #include file.

Alternatively, an error message like

cet::exception caught in art
---- Can't find key BEGIN
  fileNames 
---- Can't find key END

may be telling you that a *_module.cc file is not defined or spelled correctly. In at least one case, *_module.cc was instead misspelled as *_moduel.cc. While this module still compiled, jobs trying to access the module would end in the above 'fileNames' error.

UnknownModule

The following error message

%MSG-s ArtException:  LambdaAna:neutrinoana@Construction  28-Aug-2011 18:33:45 CDT ModuleConstruction
cet::exception caught in art
---- Configuration BEGIN
  UnknownModule ---- Configuration BEGIN
    Unable to load requested library ./lib/Linux2.6-GCC/libLambdaAna_module.so
    ./lib/Linux2.6-GCC/libPartCheckOut.so: undefined symbol: typeinfo for art::TFileService
  ---- Configuration END
  Module LambdaAna with version v0_07_04 was not registered.
  Perhaps your module type is misspelled or is not a framework plugin.
---- Configuration END

%MSG

tells you that art::TFileService is undefined. (In fact, we admit this looks a lot like the misspelled *_module.cc error we've seen elsewhere!) But, by all means, be sure you are using the current version of the CMakeLists.txt file for your module. You will want to check the list of libraries to be linked against for modules.

One of the export sub-branches is not present in the import TTree

Errors of this type are usually the result of trying to output an object from an input file whose data member list has changed since the file was created. There are two ways to configure your job .fcl file to avoid the error:

  1. Add the "outputCommands" option to the output stream configuration:
    outputs:
    {
     out1:
     {
        module_type: RootOutput
        fileName:    "tracks.root" 
        outputCommands: [ "keep *", "drop sim::Electronss_driftel_*_GenieGen" ]
     }      
    }
    

    This will configure your job to simply not write out the offending object, in this case sim::Electrons created by module labeled driftel in process GenieGen.
  2. Add the "fastCloning" option to the output stream configuration:
    outputs:
    {
     out1:
     {
        module_type: RootOutput
        fileName:    "tracks.root" 
        fastCloning: false
     }      
    }
    

    This will write the object out to the output file and add the missing data member along the way. It is a slower option than the previous option.

ProductNotFound

Errors of the form

%MSG-w AssociationUtil:  Track3DKalmanSPS:trackkal 07-Mar-2012 22:55:46 CST  run: 1 subRun: 0 event: 1 
unable to find art::Assn:
---- ProductNotFound BEGIN
 getByLabel: Found zero products matching all criteria
 Looking for type: art::Assns<recob::Prong,recob::Cluster,void>
 Looking for module label: spacepoints
 Looking for productInstanceName:

indicate that the art::Event cannot find a data product with the specified type in the module indicated by the module label. The productInstanceName is an optional additional specifier for the product and need not actually be specified.

One can verify what object types are stored in the art::Event as well as their module labels by doing

lar -c job/eventdump.fcl my_file.root

where my_file.root is the name of the ROOT file being used. Details on how to interpret the output are found here.

no ServiceRegistry has been set for this thread

Errors of the form

terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &(art::ExceptionDetail::translate(art::errors::ErrorCodes))>'
  what():  ---- NotFound BEGIN
  Service  no ServiceRegistry has been set for this thread
---- NotFound END

Aborted

indicate that there is a service that is not configured properly. This may mean that a service is being used which has not been included in the services block of the .fcl file.

It could also mean that you are attempting to use a service in the constructor of a stored object, which is a bad idea and should not be done. Avoiding this resolves the issue.

Illegal attempt to retrieve an unregistered product

These errors have messages of the form

No product is registered for
process name: 'SomeProcess'
module label: 'somemodule'
product friendly class name: 'XXX::YYY'
product instance name: ''
branch type: 'Event'

And they are telling you that you have not registered a product to be stored in the event record in the module constructor. To solve the problem, you need to put a

produces< XXX::YYY>();

line in the module constructor. If you are trying to store a collection of data products, the produces call should reflect that, i.e. for a std::vector do

produces< std::vector<XXX::YYY> >();

RNGservice problem

e.g.:
---- RANDOM BEGIN
RNGservice::getEngine():
The requested engine "t962gana:" has not been established.
cet::exception going through module T962G4Ana/t962gana run: 1 subRun: 0 event: 1
---- RANDOM END

See the top of https://cdcvs.fnal.gov/redmine/projects/larsoftsvn/wiki/Saving_and_recovering_random_number_generator_states_. In short, make sure to have createEngine() in your constructor.

Batch Submission Problems

Please see this page for trouble shooting tips.