Project

General

Profile

Fragments and FragmentGenerators w Toy Fragments as Examples

Currently being rewritten as of Mar-28-2016

n.b. Before reading this section, please first read the artdaq-demo Overview. Also potentially useful will be the UML class diagrams shown at the bottom of this page

Intro

As mentioned in this wiki's Overview, artdaq-demo provides an example of a fragment generator, ToySimulator, which developers of artdaq-based DAQ systems can study and use as a starting-off point for writing their own fragment generators. In this section of the wiki, we'll start by studying the ToySimulator code, and then later look more in-depth into the components it relies on. While reading this document, it will be helpful to look at the artdaq-core-demo and artdaq-demo source code, found in the subdirectories of the same name relative to the base directory in which you ran artdaq-demo's quick start script using the Installing and building the demo instructions. To begin with, of particular interest will be the ToySimulator code, artdaq-demo/artdaq-demo/Generators/ToySimulator.hh and artdaq-demo/artdaq-demo/Generators/ToySimulator_generator.cc as well as the interface to the (fake) hardware, artdaq-demo/artdaq-demo/Generators/ToyHardwareInterface/ToyHardwareInterface.hh, provided as an example of a vendor-supplied API to an experiment's hardware.

What code gets executed during state transitions

If you've already run the demo as described in Running a sample artdaq-demo system, you'll have a feel for the state model of artdaq-based systems; in short, the (primary) transitions they support are initialize, start, stop and shutdown. In order to write a fragment generator for an artdaq based system, you'll need to understand what parts of the fragment generator are executed during what transition:

initialize: The fragment generator object is created - i.e., its constructor is called. The constructor should take as an argument a FHiCL parameter set which will be used both to set member values in the fragment generator itself as well as its CommandableFragmentGenerator base class.

start : The fragment generator's start() function is called. An example of how this function might be used would be to use the hardware's vendor-supplied API to signal to the hardware that it can start sending data

stop : The fragment generator's stop() function is called. Here, e.g., one could signal to the hardware to stop sending data

shutdown : The fragment generator object is destroyed - i.e., its destructor is called.

A quick overview of the ToyHardwareInterface API

This API was written in C++03 to reflect that many (most?) vendor-supplied APIs in use haven't yet started using C++11. A description of the functions it provides follows; note than "hardware" below doesn't refer to literal hardware, but rather the imaginary hardware we pretend the API interfaces to:


Signal to the upstream hardware to start / stop sending data:

  void StartDatataking();
  void StopDatataking();


Allocate / deallocate the memory buffer which will be used to store the hardware's data- to be used in place of new/delete:

 void AllocateReadoutBuffer(char** buffer);
 void FreeReadoutBuffer(char* buffer);


Once the memory buffer is allocated, pass it to this function to be filled with the hardware's data; the function also returns the number of bytes read:

void FillBuffer(char* buffer, size_t* bytes_read);


In order: provide the board's serial number, the number of bits representing an ADC value, and the board type:

int SerialNumber() const;
int NumADCBits() const;
int BoardType() const;

A breakdown of ToySimulator

With a basic understanding of what parts of ToySimulator are executed at different stages of datataking and the hardware API it will be working with, we can examine its code.

Constructor:

In the body of the constructor, three main things happen:

  • a buffer (pointed to by the readout_buffer_ member) has memory allocated in which upstream hardware data will be stored
  • vendor-supplied info gets stored in the member metadata_ struct
  • Based on the vendor-supplied board type value, we set the fragment_type_
 hardware_interface_->AllocateReadoutBuffer(&readout_buffer_);

  metadata_.board_serial_number = hardware_interface_->SerialNumber();
  metadata_.num_adc_bits = hardware_interface_->NumADCBits();

  std::cout << "BOARD TYPE: " << hardware_interface_->BoardType() << std::endl;

  switch (hardware_interface_->BoardType()) {
  case 1006:
    fragment_type_ = toFragmentType("TOY1");
    break;
  case 1007:
    fragment_type_ = toFragmentType("TOY2");
    break;
  default:
    throw cet::exception("ToySimulator") << "Unable to determine board type supplied by hardware";
  }

start():

Hardware is told to start sending data

hardware_interface_->StartDatataking();

getNext()_:

(Some in-code comments stripped out for clarity)

  • If a "stop" transition has been issued to the DAQ, simply return false; we're done obtaining data
  • Otherwise, fill readout_buffer_ with data from the hardware; the hardware API provides us with the # of bytes read in
  • Use the artdaq::Fragment's FragmentBytes factory function to create a new artdaq::Fragment. In order, the arguments define the size of the fragment payload, its sequence ID, fragment ID, fragment type and the metadata. Details will be given in the next document.
  • Use the classic "memcpy" function to copy the contents of the buffer into the beginning of the artdaq fragment's payload
  • Put the artdaq fragment into the frags vector (passed-by-reference to getNext_()) and do this without unnecessarily copying memory
  • If we have a metric manager object, broadcast how many fragments have been sent
  • Increment the event counter, used to set the sequence ID in the FragmentBytes function
if (should_stop()) {
    return false;
}

std::size_t bytes_read = 0;
hardware_interface_->FillBuffer(readout_buffer_ , &bytes_read);

std::unique_ptr<artdaq::Fragment> fragptr(
                                            artdaq::Fragment::FragmentBytes(bytes_read,
                                                                            ev_counter(), fragment_id(),
                                                                            fragment_type_,
                                                                            metadata_));

memcpy(fragptr->dataBeginBytes(), readout_buffer_, bytes_read );

frags.emplace_back( std::move(fragptr ));

  if(metricMan_ != nullptr) {
    metricMan_->sendMetric("Fragments Sent",ev_counter(), "Events", 3);
  }

  ev_counter_inc();

stop():

Hardware is told to stop sending data

hardware_interface_->StopDatataking();

Destructor:

Deallocate the buffer used to hold upstream data

hardware_interface_->FreeReadoutBuffer(readout_buffer_);

The fundamental building block: artdaq::Fragment

The artdaq::Fragment's header

In any artdaq-based DAQ system, the data provided by an experiment is stored within an object of the artdaq::Fragment class. The physical representation of the data in the class is contained within a vector called "vals_"; the datatype in the vector is called "RawDataType", a typedef of a 64-bit unsigned integer found in artdaq-core's ./artdaq-core/artdaq-core/Data/detail/RawFragmentHeader.hh file. While the choice of this datatype is meant to be as transparent as possible to the end user of the artdaq::Fragment class, it will reveal itself occasionally in a "padding" effect, described later. On top of this vector of data, the artdaq::Fragment class also provides some logical structure. The vals_ vector can be thought of as consisting of an artdaq "header", containing information about the fragment itself used by the artdaq-based system, followed by experiment-specific data (such as optional user-defined metadata and physics data).

The header is defined in RawFragmentHeader.hh, and consists of two 64-bit unsigned integers containing multiple variables with one more 64-bit unsigned integer reserved for future use, organized via the following bitfield:


  RawDataType word_count          : 32; // number of RawDataTypes in this Fragment
  RawDataType version             : 16;
  RawDataType type                :  8;
  RawDataType metadata_word_count :  8;

  RawDataType sequence_id : 48;
  RawDataType fragment_id : 16;

  // 27-Feb-2013, KAB - As we discussed recently, we will go ahead
  // and reserve another longword for future needs.  The choice of
  // four 16-bit values is arbitrary and will most certainly change
  // once we identify the future needs.

 RawDataType unused1     : 16;
  RawDataType unused2     : 16;
  RawDataType unused3     : 16;
  RawDataType unused4     : 16;

The variables besides metadata_word_count and word_count (internally calculated) as well as version (referring to software version) should be set by the user. The sequence_id uniquely identifies an experiment's event, whether triggered or a timeslice, and the fragment_id uniquely identifies a subset of the full amount of data representing an event, typically corresponding to a physical subsection of a detector. For the meaning of "type", an examination of artdaq's Fragment.hh header will reveal that it defines a datatype called "type_t", an unsigned 8-bit integer whose value provides information on the type of information a fragment contains. Some of these values are reserved for system use; the range of these values are set in artdaq's RawFragmentHeader.hh file, run from 225-255, and can be employed to tell artdaq-based DAQ systems that a run has ended, etc. Values 1-224 can be assigned experiment-specific meanings by the user; e.g., in artdaq-demo "FragmentType.hh" associates type values with different kinds of CAEN boards. The metadata_word_count and the word_count are calculated internally by the artdaq::Fragment class; the metadata_word_count is the number of RawDataTypes which describe the experiment's metadata (e.g., the trigger bits, the upstream DAQ board's serial number, etc.), and the word_count is the total number of RawDataTypes in the fragment ( 3 from the fragment header + metadata_word_count + the number which make up the experiment's data). From a programmer's perspective, although the fragment's header data is publicly accessible within the RawFragmentHeader struct, best practices is to assign to either via setter functions in the artdaq::Fragment class or through one of the artdaq::Fragment constructors or factory functions.

Setting values in artdaq::Fragment

The setter functions setUserType(), setSequenceID() and setFragmentID() will modify the first RawDataTypes in artdaq::Fragment's vals_ array representing the artdaq::Fragment header. The template function artdaq::Fragment::setMetadata(const T& md) allows one to pass a user-defined datatype, T, (typically a struct) representing experiment-specific metadata which gets stored in the vals_ array beyond the header. In order to set the amount of data held by the fragment, one can call the resizeBytes() function which takes as its first argument the number of bytes to be used to hold the fragment data beyond the header and metadata, and an optional second argument providing the default value of each byte. So, for example,


struct MetadataType {
  uint16_t board_id;
  uint32_t run_number;
};

MetadataType mtype;
mtype.board_id = 867;
mtype.run_number = 3509;

// Create the fragment, using the following factory function:

//   template <class T>
//   static std::unique_ptr<Fragment> FragmentBytes(std::size_t payload_size, sequence_id_t sequence_id,
//          fragment_id_t fragment_id, type_t type, const T & metadata);

std::unique_ptr<artdaq::Fragment> toyfrag = artdaq::Fragment::FragmentBytes( 100, 1000, 1, 1, mtype);

After this code snippet is run, toyfrag's header now represents that it comes from the event with sequence ID #1000, that it's fragment ID #1 in that event, and that it's type #1 (the first allowed user type, which should have some experiment-specific meaning here). Its metadata consists of a 16-bit unsigned integer and a 32-bit unsigned integer, representing the upstream board id and the run number. Also, through the first argument to the function, its vals_ vector gets expanded to hold an additional 100 bytes of payload beyond the metadata. Note that physically, artdaq::Fragment will "pad" these quantities as its internal representation of data is in 8-byte chunks of type RawDataType, so that although 6 bytes are requested for metadata this will be rounded up to 8 bytes, and the 100 bytes requested beyond the metadata will actually result in 104 bytes being allocated; hence the physical artdaq::Fragment object will take up 24 bytes (header) + 8 bytes (metadata) + 104 bytes (payload) = 136 bytes, rather than the 130 bytes one might assume.

Other useful functions when working with artdaq::Fragment (note the "()" simply denotes a function, and doesn't necessarily imply that no arguments are supplied to this function)

resizeBytes() -- resize the payload beyond the header and metadata to contain a desired number of bytes
sizeBytes() -- returns the number of bytes representing the entire fragment
dataSizeBytes() -- the number of bytes representing the payload beyond the header and the metadata (note that this value may not match up to the argument passed to resizeBytes() due to the padding described above)

hasMetadata() -- did the user fill the fragment with metadata via setMetadata() ?
setMetadata() -- if there's no metadata in the fragment, can add it with this function. Potentially costly memory movement can result.
updateMetadata() -- for a fragment containing metadata already, overwrite its existing metadata values with new ones

dataBeginBytes() -- returns a pointer to the beginning of the section of the fragment beyond the header and metadata
dataEndBytes() -- returns a pointer to the end of the fragment
headerBeginBytes() -- returns a pointer to the beginning of the header, i.e., the beginning of the fragment

empty() -- determine if there's any data beyond the header and metadata. True if dataBeginBytes() == dataEndBytes()
isUserFragmentType() / isSystemFragmentType() -- determine whether the fragment type is user-defined or system-defined.

Note that where the artdaq::Fragment class has functions of the form stub Bytes(), it also has functions of the form stub() -- e.g., there's a function sizeBytes(), and a function size(). In this case, the functions of the form stub Bytes() should be used - as their name suggests, they work in units of bytes, while the functions of the form stub() work in units of RawDataType - so, e.g., sizeBytes() will tell you the size of fragment in bytes, while size() will tell you the size of the fragment in units of RawDataType. Generally, the functions of the form stub() are kept for backwards compatibility, and should be considered deprecated.

Using artdaq::Fragment in an experiment-specific manner: a study of demo::ToyFragment / demo::ToySimulator

n.b. To locate the relevant source code files, take a look at artdaq-core-demo/artdaq-core-demo/Overlays/ToyFragment.* for the implementation of demo::ToyFragment and artdaq-demo/artdaq-demo/Generators/ToySimulator_generator.cc for the implementation of demo::ToySimulator. It's a good idea to study these source files while reading this documentation as they contain a good deal of useful comments on implementation-specifics which are too low-level for this discussion

In order to learn how to manipulate the artdaq::Fragment's data in a manner useful to an experiment, it's instructive to look at the "Toy" classes developed for artdaq-core-demo. As they're fairly stripped down, when writing one's own experiment-specific classes, it may be a good idea to copy them and use them as a starting point, adding additional functionality where needed.

demo::ToyFragment and demo::ToyFragmentWriter

There are some aspects of an experiment's data fragment which might be of interest that aren't covered by artdaq::Fragment. For example, it's useful for an experiment to represent its fundamental unit of data as an ADC count, the size of which of course will be unknown to the artdaq::Fragment class-- so for, example, one 64-bit RawDataType might be able to represent four 16-bit ADC counts. Also, it may be that there's information of interest concerning the fragment that's not covered by the artdaq::Fragment header or the metadata structure. In artdaq-core-demo, the artdaq::Fragment overlay class demo::ToyFragment demonstrates how the data in a fragment can be organized in a real experiment; this is used by the artdaq::CommandableFragmentGenerator-derived class demo::ToySimulator in simulation. As can be seen from the ToyFragment.hh header, the demo::ToyFragment class contains within it the definition of a metadata struct :

   struct Metadata {

    typedef uint32_t data_t;

    uint32_t board_serial_number : 16;
    uint32_t num_adc_bits : 8;
    uint32_t unused : 8; // 16 + 8 + 8 == 32 bits                                          

    static size_t const size_words = 1ul; // Units of Metadata::data_t                     
  };

As well as a header struct :

   struct Header {
    typedef uint32_t data_t;

    typedef uint32_t event_size_t;
    typedef uint32_t run_number_t;

    uint32_t event_size : 28;
    uint32_t unused_1   :  4;

    run_number_t run_number : 32;

    static size_t const size_words = 2ul;   // Units of Header::data_t                     
  };

The ToyFragment::Metadata struct's bitfield contains hardware information (the serial # of the upstream hardware DAQ board from where the fragment came, as well as the number of bits composing an ADC count); the ToyFragment::Header struct's bitfield contains experiment information (the run number, the event number, and the size of the fragment, expressed by the event_size variable in units of the Header's data_t typedef ). As ToyFragments are not designed to describe data coming from actual upstream hardware, but rather, data simulated in the ToySimulator class, the issue of how to set the values in the ToyFragment::Metadata and ToyFragment::Header structs arises. While ToySimulator will be described in fuller detail below, the following code snippet in the ToySimulator::getNext_() function indicates the proper approach (please look not only at the code but also the comments):

n.b. As of Mar-24-2016, a new release of artdaq-demo, v2_06_00, uses new ToySimulator code - please contact John Freeman, , for details before this documentation gets updated


 // Set fragment's metadata                                                    

  ToyFragment::Metadata metadata;
  metadata.board_serial_number = 999;
  metadata.num_adc_bits = typeToADC(fragment_type_);

  // And use it, along with the artdaq::Fragment header information             
  // (fragment id, sequence id, and user type) to create a fragment             

  // We'll use the static factory function                                      

  // artdaq::Fragment::FragmentBytes(std::size_t payload_size_in_bytes, sequence_id_t sequence_id,                                                             
  //  fragment_id_t fragment_id, type_t type, const T & metadata)               

  // which will then return a unique_ptr to an artdaq::Fragment                 
  // object. The advantage of this approach over using the                      
  // artdaq::Fragment constructor is that, if we were to want to                
  // initialize the artdaq::Fragment with a nonzero-size payload (data          
  // after the artdaq::Fragment header and metadata), we could provide          
  // the size of the payload in bytes, rather than in units of the              
  // artdaq::Fragment's RawDataType (8 bytes, as of 3/26/14). The               
  // artdaq::Fragment constructor itself was not altered so as to               
  // maintain backward compatibility.                  

  std::size_t initial_payload_size = 0;

  frags.emplace_back( artdaq::Fragment::FragmentBytes(initial_payload_size,
                                                      ev_counter(), fragment_id(),
                                                      fragment_type_, metadata) );

  // Then any overlay-specific quantities next; will need the                                       
  // ToyFragmentWriter class's setter-functions for this                                            

  ToyFragmentWriter newfrag(*frags.back());

  newfrag.set_hdr_run_number(999);

  newfrag.resize(nADCcounts_);

Here, as the ToyFragment::Metadata is a very simple struct consisting of two unsigned 32-bit integers, values are assigned directly to the instance of this struct rather than through setter functions before the instance is passed to the artdaq::Fragment's factory function. On the other hand the ToyFragment::Header class is a bit more complex, as it needs to keep track of the size of the entire fragment. Setter functions are used to control the values; however, note that the ToyFragment class only contains const getter functions, which is all that would be needed in the case of an overlay class for an upstream data fragment in an actual running experiment. The setter functions are left to ToyFragmentWriter, which uses non-const versions of ToyFragment's data and functionality where needed to simulate data. The ToyFragmentWriter's set_hdr_run_number() function should be self-explanatory; the ToyFragmentWriter's resize() function will take as an argument a number of ADC values to be held by the fragment and in the body of the function it will then resize both the vals_ vector in the artdaq::Fragment object it references (where the data is physically stored) as well as the event_size variable in the ToyFragment's header, used to keep track of how large the Fragment is.

Please note that the size of an ADC count is typedef'd in the demo::ToyFragment declaration as a 16-bit unsigned integer, as opposed to a 32-bit unsigned integer being the size of the "data_t" datatype typedef'd in demo::ToyFragment::Header. This means that the value passed to the ToyFragmentWriter::resize() function, the number of ADC counts, will not be the same as the value returned by a call to frags.back().dataSizeBytes(); dataSizeBytes() gives us the number of bytes beyond the metadata and artdaq::Fragment header (not to be confused with the ToyFragment header!) of the artdaq::Fragment we're working on. ToyFragmentWriter::resize()'s argument also will not be the same value as the ToyFragment::Header's event_size variable, accessible via newfrag.event_size() . Please note that in both the ToyFragment class and its derived ToyFragmentWriter class the only actual member data is a reference to an artdaq::Fragment object, whose data in turn consists solely of the "vals_" vector. In other words, all writes ultimately involve manipulating the bytes which make up the vals_ vector.

demo::ToySimulator

This brings us to the demo::ToySimulator class. Derived from artdaq::CommandableFragmentGenerator, ToySimulator adds experiment-specific functionality not covered by artdaq::CommandableFragmentGenerator . The two member functions of interest in ToySimulator are its constructor and its getNext_() function override of artdaq::CommandableFragmentGenerator's getNext_() function. Looking at the constructor, we see the following:

demo::ToySimulator::ToySimulator(fhicl::ParameterSet const & ps)
  :
  CommandableFragmentGenerator(ps),
  nADCcounts_(ps.get<size_t>("nADCcounts", 600000)),
  fragment_type_(toFragmentType(ps.get<std::string>("fragment_type",  "TOY1"))),
  fragment_ids_{ static_cast<artdaq::Fragment::fragment_id_t>(fragment_id() ) },
  engine_(ps.get<int64_t>("random_seed", 314159)),
  uniform_distn_(new std::uniform_int_distribution<int>(0, pow(2,  typeToADC( fragment_type_ ) ) - 1 ))
{

  // Check and make sure that the fragment type will be one of the "toy" types

  std::vector<artdaq::Fragment::type_t> const ftypes =  {FragmentType::TOY1, FragmentType::TOY2 };

  if (std::find( ftypes.begin(), ftypes.end(), fragment_type_) == ftypes.end() )
 {
    throw cet::exception("Error in ToySimulator: unexpected fragment type supplied to constructor");
  }

}
  • nADCcounts (the number of ADC counts per fragment, obtainable via the ToyFragment::total_adc_values() function)
  • The fragment type (allowed to be "TOY1" or "TOY2", which differ in their number of bits per ADC and which will cause the constructor to throw a cet::exception if the type isn't one or the other-- see FragmentType.hh in the Overlays/ directory for a full list)
  • A random seed for the random number generator (this should not revert to the default value unless there's only one fragment per event in the simulation, otherwise every fragment will carry identical data)

The uniform_distn_ member, also initialized in the constructor, will be used to randomly create ADC values from 0 to the max ADC value with equal probability using the random number generator seeded with the random seed

The actual generation of fragments occurs in demo::ToySimulator class's getNext_() function. If a stop command has been issued, the function will exit, returning false, and the run will come to an end. Some of getNext_()'s code has been alluded to above; it will create artdaq::Fragment objects and set variables both via arguments to the artdaq::Fragment factory function, and the ToyFragmentWriter class. The arguments supplied to the artdaq::Fragment factory function could also be supplied by setter functions after the artdaq::Fragment object is default constructed; however, using the factory function which takes as one of its arguments the metadata structure means that metadata will not be accidentally neglected when formatting the fragment.

Creating your own generator and overlay classes

The source files for ToySimulator and ToyFragment can be copied and their class names changed to provide a useful starting point for creating your own simulator and overlay classes. The actual set of steps involved will be the following:

  • Copy the Toy* source code into new, appropriately-named files- both the ToyFragment* files in artdaq-core-demo's Overlays/ directory and the ToySimulator* files in artdaq-demo's Generators/ directory.
  • Search-and-replace the appropriate tokens in these copied files
  • Add the new fragment generator to the artdaq-demo/Generators/CMakeLists.txt file via the "simple_plugin" directive; simply cut-and-paste an example of this from earlier in the file and swap out the referenced fragment generator with MySimulator
  • Add the new fragment(s) to the enum list in artdaq-core-demo's FragmentType.hh and the "names" vector in FragmentType.cc . Also modify the "typeToADC" function and the constructor's "ftypes" vector in your fragment generator source file to reflect your new choice of fragment(s).
  • Recompile artdaq-core-demo and artdaq-demo
  • Test out your new fragment generator by copying the artdaq-demo/tools/fcl/driver.fcl code, editing it to use the new generator, and then calling the driver executable with it, "driver -c <yournewfcl>". The simplest way to modify your copy of the driver.fcl code is the following:
    • Replace the FHiCL code specifying the use of the ToySimulator fragment generator, i.e.,
       generator: ToySimulator
         fragment_type: TOY2        # Make sure downstream modules know about this fragment_type!
         nADCcounts: 100
         random_seed: 6514
      

      with whatever code is appropriate to the fragment generator you've written
    • Edit the "fragment_type_map" sequence in the "source" block at the bottom of the file so it contains the number and name of the fragment(s) produced by your fragment generator, where the fragment definitions will be the ones you added to FragmentType.hh and FragmentType.cc .
    • Instead of using the ToyDump art module, use the FileDumperOutput module, i.e., replace
        a1: [ toyDump ]
      #  e1: [ out1 ]
        end_paths: [ a1 ]
      }
      

      with
      #  a1: [ toyDump ]
        e1: [ out1 ]
        end_paths: [ e1 ]
      
  • Consider writing your own art module designed to analyze the data produced by your fragment generator. The simplest thing you can do is copy an existing art module (e.g., artdaq-demo/artdaq-demo/ArtModules/ToyDump_module.cc), rename it, and edit it to work with the fragments your fragment generator produces; once you've done this, assuming for the sake of argument that the name of your art module source file is "MyNewAnalyzer_module.cc", add the line
     simple_plugin(MyNewAnalyzer "module")
    

    to the CMakeLists.txt file in the ArtModules directory and run "buildtool" to compile it in. Once you've done this, you can test it out by adding it to the analysis path used in the FHiCL document you pass to the driver executable. If you wish to write more sophisticated modules, consult the links provided in art modules.

UML Class Diagrams for the demo::ToyFragment overlay class and the demo::ToySimulator fragment generator class

To learn about the syntax of UML diagrams, take a look here, or contact John Freeman, , who will be happy to give you a tutorial. The following diagrams are not comprehensive in that they don't list every function and member variable in the classes, but rather just the ones that you as a fragment generator / fragment overlay developer would care about.

UML class diagram for ToyFragment and its artdaq::Fragment object