Project

General

Profile

Feature #23286

DAQInterface should accept boot files which are in FHiCL form

Added by John Freeman 3 months ago. Updated about 2 months ago.

Status:
Reviewed
Priority:
Normal
Assignee:
Category:
-
Target version:
-
Start date:
09/17/2019
Due date:
% Done:

100%

Estimated time:
Experiment:
-
Co-Assignees:
Duration:

Description

As described in the DAQInterface Manual, the file passed to DAQInterface on the boot transition which tells it what artdaq installation to use, where to launch artdaq processes, etc., has its own formatting specifications designed to make it easy for people to read (https://cdcvs.fnal.gov/redmine/projects/artdaq-utilities/wiki/Daqinterface_boot_file_reference_). However, as FHiCL will likely continue to be used for another couple of years, it would be useful for users to to be able to provide the boot information in a FHiCL format. While DAQInterface isn't a FHiCL parser, it's likely possible to write python/sed/awk code which can approximate FHiCL parsing and thus read a FHiCL document providing the information needed on the boot transition. At this Monday's meeting it was agreed that, if possible, the input FHiCL-ized boot file should have the same format as the boot.fcl which gets saved when DAQInterface uses the database. Consequently, the resolution of this issue is contingent on the review of Issue #23260, concerning the formatting of boot.fcl.

Associated revisions

Revision b463e75d (diff)
Added by Kurt Biery 3 months ago

JCF: Issue #23286: get the new defhiclize_boot_file.awk script as far as being able to convert an artdaq_process_setting list of process tables into traditional boot file format

Revision fb66149b (diff)
Added by Kurt Biery 3 months ago

JCF: Issue #23286: similar to the previous commit, make sure the list of tables of subsystems can be defhiclized

Revision 939e0a0a (diff)
Added by Kurt Biery 3 months ago

JCF: Issue #23286: make sure that keys which have spaces (e.g., "manage processes") get properly de-FHiCLized

Revision fd02db55
Added by John Christian Freeman 3 months ago

Simply a test to see if in the previous commits the reason Issue #23286 didn't show up in the issue was because of the placement of the colon

Revision c3444c4f (diff)
Added by John Freeman 3 months ago

JCF: Issue #23286: make sure to print out FHiCL-parameters-to-be-overridden as-is

Revision 7495aa2e (diff)
Added by John Freeman 3 months ago

JCF: Issue #23286: make sure to print out FHiCL-parameters-to-be-overridden as-is

Revision de6d6ed6 (diff)
Added by Kurt Biery 3 months ago

JCF: Issue #23286: have DAQInterface interpret boot files with the extension "*.fcl" as containing boot information in the FHiCL format, rather than the traditional format as described in the Manual

Revision 2be7c49b (diff)
Added by John Freeman 2 months ago

JCF: Issue #23286: before defhiclizing a boot file in FHiCL format, first apply fhicl-dump so that defhiclize_boot_file.awk knows what to expect concerning whitespace, brace placements, etc.

Revision 4de13a9d (diff)
Added by John Freeman 2 months ago

JCF: Issue #23286: change confusingly-named defhiclize_boot_file.awk to convert_fhicl_dumped_bootfile_to_traditional_format.awk and rewrite it to understand FHiCL that's been passed through fhicl-dump rather than freshly read in from artdaq_database

Revision 75e6a4e3 (diff)
Added by John Freeman 2 months ago

JCF: Issue #23286: handle lists of more than one dimension when converting fhicl-dumped boot file to standard format

Revision 3676bc77 (diff)
Added by John Freeman 2 months ago

JCF: Issue #23286: add command-line scripts which can convert a boot file in traditional format to FHiCL format and vice-versa

Revision 74a1e979 (diff)
Added by John Freeman about 2 months ago

JCF: Issue #23286: Make sure that error messages in (de)fhiclize_boot_file.sh go to stderr and during the boot transition check that defhiclization worked before proceeding

Revision 7820571d (diff)
Added by John Freeman about 2 months ago

JCF: Issue #23286: get rid of bizarre typo almost certainly caused by my having a flat Esc key on my laptop

History

#1 Updated by John Freeman 3 months ago

  • % Done changed from 0 to 100
  • Status changed from New to Resolved

This issue is resolved at the head of feature/23286_defhiclize_boot_files, commit de6d6ed6a5c5c9a5240be95c3085e18b082aa41c. If the boot file you provide DAQInterface with for the boot transition has the extension ".fcl", rather than DAQInterface immediately parsing it as a traditional boot file (as described in the DAQInterface Manual at https://cdcvs.fnal.gov/redmine/projects/artdaq-utilities/wiki/Daqinterface_boot_file_reference_), instead it will parse it assuming it's a FHiCL file, whose format matches the format of the boot.fcl file which gets saved in the database (note that this format was recently updated - see Issue #23260).

#2 Updated by Ron Rechenmacher 2 months ago

I checked out feature/23286_defhiclize_boot_files AND merged in develop.
I looked at Bug #23260 and guessed that a boot.fcl file for demo (2x2x2) should be:

DAQ_setup_script: "/home/ron/work/artdaqPrj/demo3-develop/setupARTDAQDEMO" 

PMT_host: "localhost" 

# debug level can range from 0 to 4 (increasing order of verbosity)
debug_level: 1

EventBuilder_hosts: [ "localhost", "localhost" ]
EventBuilder_labels:["EventBuilder1","EventBuilder2"]

DataLogger_hosts:["localhost"]
DataLogger_labels:["DataLogger1"]

Dispatcher_hosts:["localhost"]
Dispatcher_labels:["Dispatcher1"]

I ran just_do_it.sh specifying the boot file as the above boot.fcl.

daqinterface seemed to start the system and run through the state transitions.
However, at the end, just_do_it.sh could not find a data file and the above .fcl file was copied to the run_records
directory under the name boot.txt.

I found artdaq-utilities-daqinterface/utils/fhiclize_boot_file.awk and defhiclize_boot_file.awk.
I used the fhiclize_boot_file.awk to generate:

DAQ_setup_script: "/home/ron/work/artdaqPrj/demo3-develop/setupARTDAQDEMO" 
PMT_host: "localhost" 
debug_level: 1

artdaq_process_settings: [ { name: "EventBuilder" label: "EventBuilder1" host: "localhost"  }, { name: "EventBuilder" label: "EventBuilder2" host: "localhost"  }, { name: "DataLogger" label: "DataLogger1" host: "localhost"  }]

That did not work either and it could not be defhiclized.

#3 Updated by John Freeman 2 months ago

The reason the FHiCL described above couldn't be defhiclized is because it doesn't satisfy what I wrote, which is that its "format matches the format of the boot.fcl file which gets saved in the database". If that had been the case, the FHiCL would have looked like this, and would have been defhiclized as expected:

DAQ_setup_script: "/home/ron/work/artdaqPrj/demo3-develop/setupARTDAQDEMO" 

PMT_host: "localhost" 

debug_level: 1 

artdaq_process_settings: [{

name: "EventBuilder" 

label: "EventBuilder1" 

host: "localhost" 

}, {

name: "EventBuilder" 

label: "EventBuilder2" 

host: "localhost" 

}, {

name: "DataLogger" 

label: "DataLogger1" 

host: "localhost" 

}] 

Now, defhiclize_boot_file.awk could be made to be more flexible if we anticipate people manually editing FHiCLized boot documents rather than just extracting them from a database, but it's worth pointing out that defhiclize_boot_file.awk will never have full knowledge of the FHiCL standard.

#4 Updated by John Freeman 2 months ago

Thinking about this a little more: it may be worth feeding boot.fcl through fhicl-dump before applying defhiclize_boot_file.awk. The logic in defhiclize_boot_file.awk would only need to be slightly modified to accommodate boot.fcl fresh from the database vs. boot.fcl after being passed through fhicl-dump, plus fhicl-dump could catch illegal FHiCL.

#5 Updated by Ron Rechenmacher 2 months ago

Starting with boot.txt in artdaq-utilities-daqinterface/simple_test_config/simple_subsystems:

/home/ron/work/artdaqPrj/demo3-develop/artdaq-utilities-daqinterface/simple_test_config/simple_subsystems
ron@mu2edaq13 :^) cat boot.txt

DAQ setup script: $yourArtdaqInstallationDir/setupARTDAQDEMO

PMT host: localhost

# debug level can range from 0 to 3 (increasing order of verbosity)
debug level: 1

Subsystem id: 1
Subsystem destination: 2

Subsystem id: 2
Subsystem source: 1

EventBuilder host: localhost
EventBuilder label: EventBuilder1

EventBuilder host: localhost
EventBuilder label: EventBuilder2

EventBuilder host: localhost
EventBuilder label: EventBuilder3
EventBuilder subsystem: 2

DataLogger host: localhost
DataLogger label: DataLogger1

DataLogger host: localhost
DataLogger label: DataLogger2
DataLogger subsystem: 2

Dispatcher host: localhost
Dispatcher label: Dispatcher1
Dispatcher subsystem: 2
--2019-10-14_17:22:48--

I modified artdaq-utilities-daqinterface/utils/fhiclize_boot_file.awk to be defhiclize-able:

/home/ron/work/artdaqPrj/demo3-develop/artdaq-utilities-daqinterface/simple_test_config/simple_subsystems
ron@mu2edaq13 :^) cat boot.txt | awk -f $MRB_TOP/artdaq-utilities-daqinterface/utils/fhiclize_boot_file.awk | awk -f $MRB_TOP/artdaq-utilities-daqinterface/utils/defhiclize_boot_file.awk 

DAQ setup script: $yourArtdaqInstallationDir/setupARTDAQDEMO

PMT host: localhost

debug level: 1

Subsystem destination: 2
Subsystem id: 1

Subsystem source: 1
Subsystem id: 2

EventBuilder host: localhost
EventBuilder label: EventBuilder1

EventBuilder host: localhost
EventBuilder label: EventBuilder2

EventBuilder host: localhost
EventBuilder label: EventBuilder3
EventBuilder subsystem: 2

DataLogger host: localhost
DataLogger label: DataLogger1

DataLogger host: localhost
DataLogger label: DataLogger2
DataLogger subsystem: 2

--2019-10-14_17:27:19--

However, note that the Dispatcher information is missing because the fundamental algorithm used by fhiclize_boot_file.awk; the same thing happens with the premodified fhiclize_boot_file.awk script. The Dispatcher info can be made to reappear if an extra newline is added to the end of the file:

/home/ron/work/artdaqPrj/demo3-develop/artdaq-utilities-daqinterface/simple_test_config/simple_subsystems
ron@mu2edaq13 :^) ( cat boot.txt; echo ) | awk -f $MRB_TOP/artdaq-utilities-daqinterface/utils/fhiclize_boot_file.awk | awk -f $MRB_TOP/artdaq-utilities-daqinterface/utils/defhiclize_boot_file.awk 

DAQ setup script: $yourArtdaqInstallationDir/setupARTDAQDEMO

PMT host: localhost

debug level: 1

Subsystem destination: 2
Subsystem id: 1

Subsystem source: 1
Subsystem id: 2

EventBuilder host: localhost
EventBuilder label: EventBuilder1

Dispatcher host: localhost
Dispatcher label: Dispatcher1
Dispatcher subsystem: 2

EventBuilder host: localhost
EventBuilder label: EventBuilder2

EventBuilder host: localhost
EventBuilder label: EventBuilder3
EventBuilder subsystem: 2

DataLogger host: localhost
DataLogger label: DataLogger1

DataLogger host: localhost
DataLogger label: DataLogger2
DataLogger subsystem: 2

--2019-10-14_17:30:59--

#6 Updated by John Freeman about 2 months ago

With commit 7820571db5d8d721690136c821acf7b9f95286c2 at the head of feature/23286_defhiclize_boot_files, the code is again ready for review. A couple of points to make since my last entry:

-DAQInterface no longer requires that the boot file, when in FHiCL form, be of the form expected when the boot file is extracted from the database. It merely needs to be legal FHiCL.
-The reason for this is that on the boot transition, it calls a script, ./bin/defhiclize_boot_file.sh, which takes as an argument the boot file in FHiCL format, passes it through fhicl-dump, and then applies convert_fhicl_dumped_bootfile_to_traditional_format.awk (formerly named defhiclize_boot_file.awk) to the result
-There's also a script people can use at the command line called ./bin/fhiclize_boot_file.sh, which takes a standard format boot file, and then puts it into fhicl-dumped FHiCL form

#7 Updated by Ron Rechenmacher about 2 months ago

pushed commits to feature/23286_defhiclize_boot_files.
This commit has modified utils/fhiclize_boot_file.awk and correspondingly modified rc/control/config_functions_database_v2.py
to generate and deal with (somewhat) "standardized fhicl-dump format".
The rc/control/config_functions_database_v2.py is backward compatible with the non-fhicl-dump format because it
was trival.
I've tested this with/without the database and with/without subsystem config and with boot.txt and boot.fcl
I'm confident with a merge into develop, but will discuss with John.

#8 Updated by Ron Rechenmacher about 2 months ago

  • Status changed from Resolved to Reviewed


Also available in: Atom PDF