Project

General

Profile

Feature #5775

Support for arbitrary, random-file access

Added by Nathaniel Tagg over 5 years ago. Updated about 2 years ago.

Status:
Feedback
Priority:
Normal
Assignee:
-
Category:
Application
Target version:
-
Start date:
03/27/2014
Due date:
% Done:

0%

Estimated time:
Scope:
Internal
Experiment:
-
SSI Package:
art
Duration:

Description

Originally raised on ART-USERS list, March 27:

Dear Artlings,

I'm attempting to use ART (namely larsoft) in a non-standard mode. I want to use the executable as a service to retrieve event data from arbitrary files and events.

That is: I want an event loop like this: {
open_file(filename);
go_to_event(event_id);
art::Event* event = get_event()

event->getByLabel(....).....

}

One way: build a service that runs in a normal event loop, but then tells the InputSource where to go next. I've looked at nutools/EventDisplay/EventDisplay_service.cc, and this gets about halfway there: it allows for navigation between events, but not between files (as far as I can tell). Also, it's not entirely clear how a service hooks into the state machine; there appears to be no documentation on designing services.

Second way: don't bother invoking the ART EventProcessor and the full-fledged state machine, but instead just start in main(), instantiate my own input source, or perhaps even RootInputFile (which is what I really want), but these rely upon art "internals" that are even less well documented.

--Nathaniel

Motivation:

The point is to run an event display. The event display in question is running (using a different access method) at http://argo-microboone.fnal.gov. It works by a user requesting a specific file and event number. (Eventually, when there is real data, there will also be a method for run/subrun/event/reco version.) The server then locates the file, and requests a process running on the server to open the file, translate the data to a custom JSON format, and return it to the browser where a set of JavaScripts do the job of actually displayng it. The system is similar to that used for MINERvA, and is detailed in a paper: http://arxiv.org/abs/1111.5315 (Nucl.Inst.Meth. 676 (2012) 44-49).

Although my current system works, there are problems with speed, robustness, and maintenance, which would be improved by actually using the framework to read the data instead of using ROOT to interrogate the Events tree directly.

Implementation notes:
I've been trying to directly instantiate RootInput or RootInpuFile myself from main(), but this is fraught with hazards due to uninitialized framework services/singletons. I've been trying to replicate my own version of EventProcessor which would do the same job, but I don't really know what I'm doing so it's hard to debug.

Chris Backhouse points out that I should simply create my own RootInput InputSource, but it's unclear to me how to create that object from main() without the framework set up around it. It occurs to me that one way forward would be to make a simple service, then inside BeginJob just goes and makes it's own InputSource independent from the EventProcessor, and captures job execution until finished. I haven't tried that yet.

Herb Greenlee pointed out that the CatalogInterface and FileTransfer service interfaces do something like this, which might be my way in.

History

#1 Updated by Christopher Green over 5 years ago

  • Status changed from New to Feedback
  • Scope changed from External to Internal

I think we'd like to change the definition of this ticket to more accurately reflect what you really want, which is (I believe) for art to provide support specifically for event displays. We'd like to discuss this with you and the other interested experiments who have raised this subject in the past, get some requirements written down and actually schedule the work. Please let us know if the slight re-targeting of this ticket is acceptable to you.

#2 Updated by Nathaniel Tagg over 5 years ago

Sure, this is fine.

In successive experiments, I've found that the problem of the event loop is actually superficial: a bigger problem it appears is that the framework does not allow successive loading of files with different data products. This is also killer for my application.

#3 Updated by Nathaniel Tagg over 5 years ago

One more note:
As a final, last-ditch hack, I tried to simply loop the core of main(): making a pset with my file, creating an event processor, running it (over 1 event) then deleting the event processor and starting over.

This fails, because ART uses a singleton (!) ProductMetaData, which has no destroyer. That is, ART explictly disallows a programmer to look at files with different data products in the same job.

I do not understand why the reading of different data products in different files would be something ART doesn't explicitly support out of the box!

So, I wander off into the sunset, the mighty walls of the framework keeping me from my damsel in distress. I will continue with my previous efforts to employ sappers to expand my secret network of tunnels under the castle.

#4 Updated by Kyle Knoepfel over 4 years ago

  • Target version set to 521

#5 Updated by Kyle Knoepfel about 2 years ago

  • Target version deleted (521)


Also available in: Atom PDF