Package splitting ideas

Reacting to package splits (27-Aug-2014)
Package splitting progress (26-Aug-2014)

18-Jun-2014, KAB - In several experiments that currently use artdaq, there is some rather unfortunate coupling between the online and offline packages. Some of the negative aspects of this coupling include the following:
  1. installations of the offline software need to include software packages that are only used in the online. This causes extra work for experimenters who are maintaining software installations on analysis computers.
  2. an upgrade to the version of art that is used in the offline necessitates an update to a newer version of artdaq. This means that artdaq is occasionally forced to upgrade to a newer version of art sooner than would be convenient.

Coupling between the artdaq "data" classes and art

In order to get the most benefit from separating the online- and offline-specific parts of artdaq-based experiment code, the current artdaq code itself should be split into two or more packages. The most important part of this split is to move the "data" classes in artdaq into their own package. These classes are ones like artdaq::Fragment and artdaq::RawEvent.

One complication with this plan is that the definition of the art/ROOT data products that are built on top of these classes (e.g. art::Wrapper<std::vector<<artdaq::Fragment>>) will naturally also move to the artdaq "data" package. This means that the new artdaq "data" package will need to depend on art instead of being a bit more independent.

Internally, we've discussed the possibility of moving the necessary art classes into a separate package and thereby avoiding a dependence of artdaq-data (or artdaq-core) on the full art suite, and we don't believe that this is feasible, or strictly necessary, at this time. If such a split within art happens at some point in the future for other reasons, we should update the artdaq packages accordingly.

Proposed artdaq changes

Here is the proposed scheme for splitting up artdaq.

[28-Jul-2014, KAB - For simplicity, I simply split artdaq into artdaq-core and artdaq, for now. If we find that the generators package is needed later, we can add it back then. I should say that I did create the artdaq-generators repository before deciding on the simpler option, so it will be ready to use if we need it.]

The new packages would be the following (better names welcome):
  • artdaq-core
    • would include the classes that currently live in artdaq/artdaq/DAQdata and aren't related to fragment generation (e.g. Debug.hh/cc, features.hh, Fragment.hh/cc, Fragments.hh, RawEvent.hh/cc)
    • would also include classes from artdaq/artdaq/DAQrate that might be useful outside of artdaq (e.g. EventStore, ConcurrentQueue, GlobalQueue, MonitoredQuantity, StatisticsCollection)
    • would depend on art (and cetlib, fhiclcpp, etc. by extension)
  • artdaq-generators
    • would include the classes that currently live in artdaq/artdaq/DAQdata and are related to fragment generation (e.g. FragmentGenerator.hh/cc, GeneratorMacros.hh, and makeFragmentGenerator.hh/cc. CommandableFragmentGenerator.hh/cc would need to be discussed...)
    • would depend on artdaq-core (and art, etc. by extension)
  • artdaq
    • would contain all of the code that is currently in artdaq that hasn't been moved to a new package
    • would depend on artdaq-generators and artdaq-core (and art, etc. by extension)

In an earlier version of this proposal, artdaq-core was split into two packages: artdaq-data and artdaq-utils. If at some point, artdaq-core no longer needs to depend on the full art suite, then this option could be revisited.

Effect on experiment packages


With the artdaq package structure listed above, the DarkSide online and offline packages could be restructured in the following way:
  • darksidecore
    • would contain the data format, database interface, and online monitoring classes that currently exist in ds50daq (e.g. Config, V1190Fragment, V1495Fragment, V172xFragment, DBInterface, ds50db_interface, WFViewer, etc.)
    • would depend on artdaq-core and pqxx (and postgresql and art, etc. by extension)
  • ds50daq
    • would contain all of the existing code that is not moved into a new package
    • would depend on darksidecore, artdaq, and the CAEN libraries (and art, pqxx, etc. by extension)
    • (ds50daq currently has an explicit dependence on XMLRPC; I believe that this can be removed since we get XMLRPC from artdaq)
  • darkart
    • would contain the same code that it does now
    • would depend on darksidecore (and art, pqxx, etc. by extension)

In an earlier version of this proposal, the data format classes and the online monitoring classes were split between two packages. This proposal was based on the supposition that we could avoid having the artdaq-core package depend on art. In that case, the darksidecore package would not need to depend on art, and only the darkmon package would need to do so. However, since artdaq-core depends on art in this current proposal, it doesn't make much sense to separate the different types of core DarkSide code.

Some questions regarding this scheme:
  • Is it desirable to move the online monitoring classes from ds50daq into darksidecore, as shown in this proposal?


With the package restructuring listed above, the NOvA DDT could simply depend on the following packages:
  • artdaq-core and artdaq-generators (and art, etc by extension)


This proposed scheme does not provide any advantages to uBooNE DAQ. If artdaq-core did not depend on art, then uBooNE DAQ could use artdaq-core (or artdaq-utils and artdaq-data) and get the classes that it needs without needing to install art.

Mu2e DAQ

  • mu2edata
    • would depend on artdaq-core (and art by extension)
  • mu2e-artdaq
    • would depend on mu2edata and artdaq
  • <mu2e offline code>
    • would depend on mu2edata, etc.


  • lbnedata
    • would depend on artdaq-core (and art by extension)
  • lbne-artdaq
    • would depend on lbnedata and artdaq
  • <LBNE offline code>
    • would depend on lbnedata, etc.

Summaries of what is currently being used


[ds50gpvm01 darkart]$ findc ds50daq | grep include | cut -d':' -f2 | sort -u
#include "ds50daq/DAQ/Config.hh" 
#include "ds50daq/DAQ/V1190FragmentWriter.hh" 
#include "ds50daq/DAQ/V1495Fragment.hh" 
#include "ds50daq/DAQ/V172xFragment.hh" 
#include "ds50daq/online/ArtServices/DBInterface.hh" 
#include "ds50daq/online/ArtServices/ds50db_interface.hh" 


[novadaq@novadaq-ctrl-master DDTproto]$ findc artdaq | grep include | cut -d':' -f2 | sort -u
#include "artdaq/DAQdata/Debug.hh" 
#include "artdaq/DAQdata/FragmentGenerator.hh" 
#include "artdaq/DAQdata/Fragment.hh" 
#include "artdaq/DAQdata/Fragments.hh" 
#include "artdaq/DAQdata/GeneratorMacros.hh" 
#include "artdaq/DAQdata/makeFragmentGenerator.hh" 
#include "artdaq/DAQdata/RawEvent.hh" 
#include "artdaq/DAQrate/GlobalQueue.hh" 
include SoftRelTools/


uBooNE DAQ code currently uses the following classes (originally from artdaq, but now copies):
  • Fragment.hh/cc
  • Fragments.hh
  • ConcurrentQueue.hh
  • GlobalQueue.hh/cc
  • EventStore.hh/cc
  • RawEvent.hh/cc
  • detail/RawFragmentHeader.hh
  • features.hh


NOvA DDT requests an upgrade to a new version of art

We would create new versions of artdaq-core and artdaq-generators that depend on the new version of art and make them available to NOvA.

Mu2e offline wants to move to a newer version of art; mu2e online needs to wait for a month or so before upgrading

We would create a new version of artdaq-core that depends on the new version of art and make it available to the mu2e offline group. artdaq would be upgraded to depend on the new art, as time permits, and the new artdaq version would be made available to mu2e-artdaq when it becomes available.