Project

General

Profile

Idea #841

pre-load ET system & event builder in run series?

Added by Jeremy Wolcott almost 9 years ago. Updated over 8 years ago.

Status:
Feedback
Priority:
Low
Assignee:
-
Target version:
-
Start date:
01/05/2011
Due date:
% Done:

0%

Estimated time:
Duration:

Description

We lose a bit of beam every time a subrun turns over due to the set-up time for ET & the event builder. With the current length of our subruns, it's about a 1% loss (about 5 beam triggers every subrun, which contains ~350 beam gates for mixed mode and 700 gates for pure beam runs): which, over the course of a couple years' continuous running amounts to several weeks' worth of data. (This is consistent with Howard's AEM reports, which average around 99% live time when everything's going smoothly.) Small, but not totally insignificant.

The only real way around this I can think of is to pre-load the ET system & event builder processes. The simplest way to do this, I guess, would be to preload all of the ET processes for a run series at the beginning. (Or, more precisely, preload all but the first during the first subrun, I guess. This would have to come with some special dispensation for what happens if the next subrun starts before its ET file is ready...) The ET system creates a memory-map file the size of the entire subrun, which is on the order of a couple gigabytes, but we have plenty of disk (and these files are cleaned every day) so that's not a big problem. We'd need to preload the DAQ process too, though, I think--it needs to set up its ET system for the event builder to attach to--and this may be fraught with peril. At the very least we'd need to engineer the DAQ to wait for an external signal before trying to talk to the hardware.

It also briefly crossed my mind that we might be able to at least drive the dead time down by increasing the subrun length: fewer subrun turnovers => fewer triggers lost. I don't have any idea how the ET system/event builder/DAQ setup time scales with the number of ET "events" in a subrun, though. But maybe this might be worth playing with.

History

#1 Updated by Gabriel Perdue almost 9 years ago

It sounds like a pretty good idea. One could certainly start the ET system well before the DAQ and then tweak the DAQ to block or wait around for a while if it somehow got started before the ET system finished setting up.

#2 Updated by Gabriel Perdue almost 9 years ago

Hmm, thought I was clicking "Preview." At any rate, I also meant to say the bulk of the time for ET is spent setting up the ET system file. We pass the name of that system file to the DAQ to start things up. If we started setting the file up a couple of minutes before the end of a subrun, we could probably get about 75% of the total available improvement here. The issues are just (a) knowing when the subrun will actually stop (we could set runs to go by time instead of gates though if we wanted to), and (b) having a graceful way to change our mind. Also a small issue is that thus far we have included the timestamp of the start of setting up the ET system in the DAQ file name. This therefore tracks within a couple of minutes of the actual start of data taking. Pushing that time back could cause some problem, but probably not a meaningful one (each trigger is timestamped anyway).

#3 Updated by Jeremy Wolcott almost 9 years ago

Gabriel Perdue wrote:

Hmm, thought I was clicking "Preview." At any rate, I also meant to say the bulk of the time for ET is spent setting up the ET system file. We pass the name of that system file to the DAQ to start things up. If we started setting the file up a couple of minutes before the end of a subrun, we could probably get about 75% of the total available improvement here. The issues are just (a) knowing when the subrun will actually stop (we could set runs to go by time instead of gates though if we wanted to), and (b) having a graceful way to change our mind. Also a small issue is that thus far we have included the timestamp of the start of setting up the ET system in the DAQ file name. This therefore tracks within a couple of minutes of the actual start of data taking. Pushing that time back could cause some problem, but probably not a meaningful one (each trigger is timestamped anyway).

I did think about doing the pre-loading towards the end of the previous subrun, but the issue associated with that is that the user would then not be allowed to use the "skip to next subrun" button until the ET system is loaded. I suppose this could be modified to support a complimentary paradigm instead, where if the 'skip' button is pressed or the subrun turns over before ET is ready, then it does setup on an as-needed basis.

To address your points above:
(a) I wouldn't mind transitioning to a fixed-time subrun scheme, though of course that would mean a fair number of modifications to our current run series architecture (which would mean a lot of mindless code-sprucing by me) which I am not so excited about. I suppose we could instead use a configurable fraction of the number of gates as a threshold: start loading the next ET system when the subrun is 75% complete (where 75% is a configurable parameter).
(b) It's easy enough to make this "pre-loading" a flag in the configuration. In any case it will need to be 'switchable' because there's no way I would want it running that way all the time on the test stand.

I'll keep ruminating on it.

#4 Updated by Jeremy Wolcott over 8 years ago

  • Priority changed from Normal to Low


Also available in: Atom PDF