SAM Process ID in metadata
Robert Illingworth pointed out that it would be helpful to store the SAM process ID in the file metadata. It would be nice if ART could pick this up and add it. NOvA could build this into it's own metadata module, but Robert thought it is an appropriate all-experiment feature. The art_sam_wrap.sh shell script stores the process id in a variable called $consumer_id, so the art code could try to key off that. Otherwise, it seems appropriate to pass it in as an environment variable, art_sam_wrap.sh could be modified to pass it in.
#1 Updated by Christopher Green about 5 years ago
- Status changed from New to Feedback
According to the code, the
services.FileCatalogMetadata.processID parameter (settable via the
--sam-process-id option) is already added to the file catalog metadata as
process_id. Please let us know whether this item is present in the output of
sam_metadata_dumper for an output file that you know used SAM as input, as the absence of this item would indicate a bug which should be found and fixed.
#2 Updated by Dominick Rocco about 5 years ago
It appears that the --sam-process-id flag can only be passed in the presence of --sam-web-uri. In our case, art_sam_wrap.sh does the fetching of the file, so we don't pass in --sam-web-uri. I think that the desired behavior is to require the process ID in the presence of a web URI, but not vice versa. Is that a sensible alternative? Here's the output I see when --sam-web-uri is not included.
$ nova -c /grid/fermiapp/nova/novaart/novasvn/releases/development/ProductionScripts/fhicl/prod_reco_numi_job.fcl /nova/ana/users/rocco/tmp/neardet_genie_fhc_nonswap_geantonly_1000_r01000095_s54_S14-08-01_v1_20140620_170832_fnpc2058.fnal.gov_1404203105_4746_0_sim.pid.root --sam-process-id 99 OptionsHandler caught a cet::exception calling art::FileCatalogOptionsHandler::doProcessOptions() ---- Configuration BEGIN configurations services.user.CatalogInterface.webURI (--sam-web-uri) and services.FileCatalogMetadata.processID (--sam-process-id) must be specified together or not at all. ---- Configuration END Art has completed and will exit with status 7002.
When I do pass them together, I get a different exception, but it may be simply because of a poorly formed URI when trying to run on a local file.
$ nova -c /grid/fermiapp/nova/novaart/novasvn/releases/development/ProductionScripts/fhicl/prod_reco_numi_job.fcl /nova/ana/users/rocco/tmp/neardet_genie_fhc_nonswap_geantonly_1000_r01000095_s54_S14-08-01_v1_20140620_170832_fnpc2058.fnal.gov_1404203105_4746_0_sim.pid.root --sam-process-id 99 --sam-web-uri stuff --sam-data-tier=out1:pid --sam-stream-name=out1:0 %MSG-s ArtException: nova 01-Oct-2014 09:27:25 CDT JobSetup cet::exception caught in art ---- OtherArt BEGIN ServiceCreation ---- Type mismatch BEGIN processID ---- Type mismatch BEGIN can't obtain atom from sequence ---- Type mismatch END ---- Type mismatch END cet::exception caught during construction of service type art::FileCatalogMetadata: ---- OtherArt END %MSG Art has completed and will exit with status 1.