Getting DAQInterface to manage memory

When transferring fragments between artdaq processes, these processes need to be told via FHiCL documents how large the memory buffers designed to hold fragments need to be. A large experiment which uses a fragment generator to read out tens of thousands of ADC counts in a single event will clearly have different memory needs than a small teststand designed to study a hardware component which only produces a few bytes containing a timestamp and a checksum in each event; experimenters will want to make sure there's enough space allocated for the movement of their data without unnecessarily wasting resources. There are two options for how DAQInterface manages memory, one of which is more crude but only requires setting one value, another of which is more sophisticated but requires values to be set in the FHiCL configuration documents which DAQInterface is given at the start of the config transition. You can toggle this option by either setting "advanced_memory_usage: false" or "advanced_memory_usage: true" in the DAQInterface settings file. Details are as follows:

Non-Advanced Memory Usage

This is the traditional manner in which DAQInterface handles allocating memory buffers for data transfer. It uses the value of the max_fragment_size_bytes parameter in the settings file to determine how much space to allocate for each fragment. E.g., if this value is set to 10000000 (10 megabytes) in the settings file, then in a run which uses five boardreaders each sending one fragment per event, the buffer size for the transfer plugins between boardreaders and eventbuilders will be 10 megabytes, and the buffer size for the transfer plugins between eventbuilders and dataloggers, or dataloggers and dispatchers, will be 50 megabytes, since the entire event needs to be transferred between those processes. However, if the FHiCL document for one of these processes has the parameter max_event_size_bytes set, then the value of that parameter will override the product of the number of fragments per event times the settings file's max_fragment_size_bytes; this can be useful if there's a large variance in the actual maximum fragment size the boardreaders send. Note that if max_event_size_bytes doesn't appear in the FHiCL document, then max_fragment_size_bytes needs to appear, otherwise the artdaq processes won't initialize successfully and DAQInterface will be unable to enter the Ready state.

Advanced Memory Usage

When advanced memory usage is set in the DAQInterface settings file, a few new requirements are brought in, since DAQInterface needs to know what the max fragment size is from each boardreader. The advantage of this is you don't end up in situations where you have a 100 megabyte buffer transferring a payload of just a few bytes, but the disadvantage is that the FHiCL configurations need to be specially edited to support this feature, as you'll see in a moment. The requirements are:

  • max_fragment_size_bytes must not be set in the DAQInterface settings file, as setting this is logically exclusive of the behavior of advanced memory usage. It's recommended that you comment it out (as opposed to deleting it), in case you wish to switch back to non-advanced memory usage.
  • Instead of being set in the settings file, max_fragment_size_bytes must be set in the FHiCL document for every BoardReader used in a run. E.g., in a BoardReader's ToySimulator FHiCL, you might see something like:
    nADCcounts: 5000
    max_fragment_size_bytes: 10200

    where you want to ensure that max_fragment_size_bytes represents the largest fragment you can imagine your fragment generator producing. In the above example, the ToySimulator is configured to produce 5000 2-byte ADC counts per fragment; setting the max fragment size to 10200 bytes more than allows for the additional space required by the artdaq fragment header. As is the case with max_fragment_size_bytes in the settings file, hex values are supported.
  • The FHiCL document for every other type of artdaq process (i.e., the non-BoardReaders) must not contain a max_fragment_size_bytes variable. The reason is that this variable is ignored by DAQInterface when advanced memory usage is in place and therefore would be highly misleading if found in non-BoardReader processes.
  • max_event_size_bytes must not appear anywhere in the FHiCL documents; this is because the point of advanced memory usage is that the max event size is calculated off of the individual boardreaders' max fragment sizes

When allocating buffer sizes, DAQInterface will use the value of max_fragment_size_bytes as found in a boardreader's FHiCL document for the buffer for transferring data between boardreaders and eventbuilders. When doing this for entire events between eventbuilders and dataloggers, or dataloggers and dispatchers, it will use the sum over the max_fragment_size_bytes from each boardreader with a 10% overhead added to account for additional art info which gets added to the byte stream. Related to this is that the max_event_size_bytes parameter in a FHiCL document will have its value overwritten by DAQInterface before it's sent to the artdaq process.

If a boardreader isn't designed to send one fragment per event

While boardreaders typically send one fragment per event, and this is what's assumed by default when DAQInterface allocates buffer sizes, this isn't necessarily always the case. Some boardreaders which are used just for their functionality during configuration may send no fragments, while others may send multiple fragments. DAQInterface needs to be told this in order to properly calculate buffer sizes; so, e.g., if a given boardreader sends 10 fragments per each event, the following parameter will need to be added to the boardreader FHiCL document:

generated_fragments_per_event: 10

If this parameter isn't set, then there very likely won't be enough memory to transfer events between artdaq processes in this particular example, and backpressure will occur.