Cross-platform end-of-line format
Now that Mac platform is supported, I have started receiving reports of cryptic errors like:
Failed to parse the configuration file 'hack_evd.fcl' with exception
---- Malformed #include directive: BEGIN
at line 1 of file ./hack_evd.fcl
---- Malformed #include directive: END
It was due to the fact that the termination of the
#include line was
"\x0A\x0D" and the parser was not recognizing it.
I suspect that that FHiCL file was created on a different platform (I actually think
"\x0A\x0D" is DOS/Windows format) and copied it to a different machine... or whatever.
Also, it seems it's a problem specific to the
#include directive or, at least, not general.
The point is, to find what's the problem there is sure source of grief for most people.
#3 Updated by Gianluca Petrillo over 4 years ago
- File DumpWires_dos.fcl DumpWires_dos.fcl added
- File DumpWires_mac.fcl DumpWires_mac.fcl added
- File DumpWires_unix.fcl DumpWires_unix.fcl added
Mirabile dictu, I can.
The following test is run on a SLF6 machine (uboonegpvm06.fnal.gov).
I have saved the same file with KDE editor,
kate, using three different End-Of-Line styles.
The first line of the FHiCL file is just a hash symbol...
Running on a UNIX-style EOL:
$ hexdump -C DumpWires_unix.fcl | head -n 1 00000000 23 0a 23 20 46 69 6c 65 3a 20 20 20 20 20 44 75 |#.# File: Du| $ lar --debug-config test.cfg -c job/DumpWires_unix.fcl ** ART_DEBUG_CONFIG is defined: config debug output to file test.cfg ** Art has completed and will exit with status 1.
As you can see,
hexdumpshows that the termination line (second character in the file) is
The parser is happy.
Running on a Mac-style EOL:
$ hexdump -C DumpWires_mac.fcl | head -n 1 00000000 23 0d 23 20 46 69 6c 65 3a 20 20 20 20 20 44 75 |#.# File: Du| $ lar --debug-config test.cfg -c job/DumpWires_mac.fcl Failed to parse the configuration file 'job/DumpWires_mac.fcl' with exception ---- Parse error BEGIN Local lookup error ---- Can't find key BEGIN microboone_geometry_helper (at part "microboone_geometry_helper") ---- Can't find key END at line 1, character 375, of file "./job/DumpWires_mac.fcl" ExptGeoHelperInterface: @local::microboone_geometry_helper # defined in geometry_microboon Geometry: @local::microboone_geo # defined in geometry_microboon} # physics: [ ana ] "daq"ata"sed); in the comments, defaults are reportede) ^ ---- Parse error END Art has completed and will exit with status 7002.
Here the termination is
"\x0D", that FHiCL parser hates but only in the
If I place a UNIX termination line before the
#includeline, or if the
#includeline is the first in the file, I will get a different error (
Malformed #include directory). If I add also a UNIX terminator at the end of the
#includeline, it becomes content.
Finally, running on a DOS-style EOL:
$ hexdump -C DumpWires_dos.fcl | head -n 1 00000000 23 0d 0a 23 20 46 69 6c 65 3a 20 20 20 20 20 44 |#..# File: D| $ lar --debug-config test.cfg -c job/DumpWires_dos.fcl ** ART_DEBUG_CONFIG is defined: config debug output to file test.cfg ** Art has completed and will exit with status 1.
Here the termination is
"\x0D\x0A", and everything is well.
I suspect that
"\0d" does not break a line, but since
"\0d" is still considered a blank character and FHiCL is mostly able to be written on a single line, that typically does not matter. I am unsure how comment parsing enters this explanation though.
Also note that
#include lines are already special in that they can't accept a comment at their end (I am tempted to call this a bug).
The original configuration files I used are attached and, in case the uploading messes with EOL terminations, also available in
/uboone/data/users/petrillo/LArSoft/develop/prof/job (accessible from any MicroBooNE GPVM, e.g.
#4 Updated by Kyle Knoepfel over 4 years ago
- Status changed from Feedback to Assigned
- Estimated time changed from 2.00 h to 4.00 h
Thank you for the configuration files. We now understand what the problem is. In order to insert "#include" files into the final FHiCL file, we depend on
std::getline, whose default delimiter is
'\n'. For the UNIX and DOS format, both include the
'\n' to signify the end of a line (DOS files also include
'\r' before EOL). However, for older-style Mac file formats, the newline is signified only by the carriage return
'\r'. The fix is relatively straightforward and almost fully implemented.