Project

General

Profile

Feature #1758

Histograms in shared memory

Added by Mark Messier about 8 years ago. Updated about 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
I/O
Target version:
-
Start date:
09/01/2011
Due date:
% Done:

0%

Estimated time:
Scope:
Internal
Experiment:
NOvA
SSI Package:
art
Duration:

Description

I am developing software for monitoring data quality from out DAQ. The process will look at events with a very high duty factor and make sets of summary plots. As the producer job must run with high duty cycle, the application is split into two parts: a producer that makes and fills the histograms and a viewer that shows the histograms to the shifter who views them in the control room. Communication of the histograms between these two is accomplished in my prototype by storing them in shared memory using the ROOT TMapFile class. It would be very convenient if the ART framework provided a mechanism for sharing histograms between two jobs via shared memory or by some other means.


Related issues

Related to art - Bug #1340: Allow creation of subdirectories with TFileServiceClosed06/04/2011

Related to art - Feature #1316: Snapshot histogram fileFeedback05/19/2011

Related to art - Feature #1759: Periodic refreshing of histogram filesAccepted09/01/201109/30/2013

History

#1 Updated by Walter E Brown almost 8 years ago

  • Status changed from New to Accepted

Should be considered in context of the intended TFileService review.

#2 Updated by Mark Messier over 7 years ago

I was wondering if I could get a status update on this? We are actively developing the NOvA online monitoring using ROOT's TMapFile and link to ROOT's -lNew. The interaction between this and ART seems unstable: Updates to the TMapFile regularly seg. fault in the ART context but similar usage in bare ROOT sessions are stable. If this service were available in ART now we would start using it today.

#3 Updated by Mark Messier over 7 years ago

Any updates on this request? We are rolling out our OnlineMonitoring in the next few weeks and could really use this.

#4 Updated by Christopher Green over 7 years ago

An email I sent 2012/07/02:

On 6/30/12 2:28 PM, Mark Messier wrote:

Hi Chris,

I was wondering if there has been any progress on sharing root histograms via shared memory (or by any other mechanism). My student and I have been developing the new ART based online monitoring and are at the point where we need this. We tried using roots MemoryMap but linking to libNew reliably causes seg. faults as you warned. Right now we are using disk resident files to exchange histograms but that has very high latency.

Thanks for any info you can provide.

Hi Mark,

Coincidentally, we had an all-day meeting last week to understand current and outstanding art tasks both for mainstream art and our "artdaq" efforts. After further discussing with Marc, we think that a naive implementation of (eg) TH1's in shared memory would almost certainly have many problems and at the least be very fragile. With that in mind, I think it would be very useful for us to get together and understand exactly what it is that you want from this feature in terms of functionality so we can come up with a solution for you that not only does everything you need it to do, but also isn't fragile or too complex due to having unnecessary features or flexibility.

I have cc:'d the artists list because coming up over the next few weeks are several vacations within our group, some of which overlap: Jim is out this week, I am out from Wed through the end of next week, and Marc leaves toward the end of net week for a couple of weeks. In general, but certainly over Summer, it is to your advantage to correspond with us via the artists or art-users lists or using the issue tracker: that way someone should see and respond to your message.

I have a couple of questions for you to consider until we get to have our meeting to enable us better to work towards something that will satisfy your needs:

  1. What is the priority of this relative to the EventDisplay work and the ability to decide whether to run an analyzer based on filter results?
  2. One consumer or multiple?
  3. Constant access to coherent data or periodic?
  4. What data do you need to share in this way? Just histograms or other things in addition? Shared memory has severe restrictions on what can be placed there (no pointers, no polymorphic classes, for instance), so almost any ROOT entity will need some kind of special handling, such as serialization. It's possible however, that there is a solution which fits your needs which isn't shared memory in this case.

Thanks,
Chris.

<snip>

#5 Updated by Mark Messier over 7 years ago

Hi Chris,

Here are my answers and a few additional questions:

1. We will need this by about November of this year when we will be bringing up the detector. I would rank it higher than the event display which has been in its current state for so long that we've gotten used to it (although we would still very much like to have real interactive capabilities) and the filter which has work-a-rounds in the current framework.

2. Multiple

3. Not sure what this is asking. I guess the answer is periodic, but the frequency of access could be high and the latency must be much better than disk access which is what we are using now as a work-a-round.

4. For the monitoring I would like to push histograms and a small custom ROOT object summarizing the producer status.

I'm open to non-shared memory solutions, but as it happened there was a shared memory solution built into ROOT which we used to be able to use out-of-the-box which met our needs; hence my framing of the feature in terms of shared memory.

#6 Updated by Marc Paterno about 7 years ago

  • Priority changed from Normal to High

#7 Updated by Mark Messier almost 7 years ago

The priority of this can be downgraded. I've implemented my own solution which which allows a histogram server and client to exchange histograms via shared memory.

#8 Updated by Christopher Green over 6 years ago

  • Tracker changed from Feature to Bug
  • Status changed from Accepted to Feedback
  • Priority changed from High to Normal

#9 Updated by Christopher Green over 6 years ago

  • Tracker changed from Bug to Feature
  • Category set to I/O
  • Scope set to Internal
  • Experiment NOvA added
  • SSI Package art added
  • SSI Package deleted ()

Please advise if this feature is still requested, or if the uesr-side solution is acceptable.

#10 Updated by Kyle Knoepfel over 4 years ago

  • Target version set to 521

#11 Updated by Kyle Knoepfel almost 3 years ago

  • Status changed from Feedback to Closed

Closed due to lack of feedback.

#12 Updated by Kyle Knoepfel about 2 years ago

  • Target version deleted (521)


Also available in: Atom PDF