Tasks for DAQ commissioning by 30-Sep-2014, revisited 9-July-2014¶
Things accomplished¶(1) Triggered data readout works in non-compressed mode for the TPC crates for short windows (33 ticks, as opposed to 3200 ticks). That works on an internally-driven software trigger. Calibration mode data taking works for TPCs: ASICs configured, pulser configured, channel mapping confirmed. [MRT test stand]
(2) The TPC crate can also run in this mode for the full sized 3200x3 windows and with an external trigger at 4 Hz. [DAB Annex test stand]. Namely we can take triggered data.
(3) The PMT Trigger Board can be properly triggered on receipt of an external pulse. It writes out the expected 1500- and 20-sample (tunable) packets, per discriminator settings/ext-pulse. It can do this at a satisfactory rate (~100Hz, much higher than BNB/NuMI rate). Flasher mode data taking works for PMTs: LED pattern flashes, PMT mapping confirmed. [MRT+ test stand]
(4) assembler can run in a mode where it reads out from both crates and writes data to disk.
(5) The message passing works to allow a script to step through runs.
(6) The DAQ computer cluster is up at LArTF, and driver (and in fact all DAQ base level) code works there -- once PCIe cards are stuffed so that we may take the DAQ out there.
(7) Fanned out trigger and clock out to two crates
(8) Online monitoring works. Connects to the shared memory segments -- one on assembler, the other on a seb in "dispatcherToSHM" thread. Can scale to 10 sebs trivially. ubdaq-prod-near1 hosts apache server port access, etc. Integrated now into RunControl-- meaning ProcessManager startup/heartbeat monitoring.
(9) State machine finalized, and all elements of it incorporated into the code/command line routines (e.g., moving into a shutdown state that properly kills the sebApp and assemblerMain processes). - As part of this, the shm segments easily reattached after runs end. No more cumbersome ipcrm'ing.- This last bit is fixed by virtue of Nathaniel switching the implemented shm segments to his own, not Gennadiy's, which allow to reattach to an in-use segment.
(10) Solved: Why is readFragment taking so long to read its last data card from circularBuffer? Why does fragmentRate look 10x lower than networkRate on ganglia? Size meant number of 64 byte words in artdaq, surprisingly. Are fragments 10x bigger than we think? No. Code run time puts our limit at 90 Hz max, which is sufficient now.
(11) A condor system is installed on LArTF cluster
(12) Fake data mode running to allow development on LArTF cluster. Meaning, sebApp, assemblerMain run creating "real-" looking data, dispatcher attaches to shm, memcopies to another shm, and ommain runs and attaches to that shm and produces .root files.
(13) ubdaq-prod-smc physically at LArTF and networked, dBs largely exist.
(14) The trigger development mature. Have run with the real beam triggers (not yet the external strobe) developing the timing and NIM logic.
(15) GPS antenna installed on LArTF roof
(16) Run control works. Our experts run the rc build. Scripting and configurations can run ASICS calibration
- The Glassman cathode HV is controlled through the Glomation from python script on uboonedaq-smc.
- The BiRA PMT HV crate works under control of EPICS CSS GUI and VxWorks controller.
- The Weiner MPODs for ASICs power and wire bias are under EPICS moncon.
- The Weiner PL508s are under EPICS moncon.
- Much data -- IFBeamsdB devices, cryo data -- is monitored/displayed via EPICS CSS displays
- EPICS IOCs do the moncon of fanpacks/RTDs through the Glomations
- server CPU monitoring, fanspeeds monitored by IPMI
- The Glomations boot desired IP addresses with a kernel and a filesystem they get from local, individually created SDRam cards.
- data and network fibers, trunk lines and panel installed.
Things to Do¶
(0) Do we still have license issues with windreg drivers for PCIe cards? Comes and goes with reboots, it seems. Contact Jungo, if so.
Gennadiy, Eric, Kazu
(1) ? The trigger board in the PMT crate does not yet provide an output pulse upon instruction to do that from the software (Internal trigger mode). That must work to truly run the ASICs calibration mode on all TPC crates. It also must work to run the flasher system as desired. Pulser needs to physically go to LArTF, perhaps by end of January. !!This may be done, will check!!!
(2) State machine needs bumps still smoothed out. Practice on DAB Annex. Introduce mf::logger logging to files/a viewer.
Gennadiy, Eric, Wes
(5) Need to build the fhicl files on the fly from the DAQ runConfig dB. FPGA file names need to be put into fhicl files.
Wes, Andrzej, Eric, David K
(6) Problem of end-of-event missing word (sometimes beginning-of-event) caused by apparent race condition still elusive -- needs to be understood. Seems to allow running, nevertheless for 10s of minutes.
Eric, Wes, Georgia
(7) Integrate GPS information into each cardHeader. Code just needs uncommenting in sebs/rawFragmentDMASource::readFragment().
(8) We need process to JSON-ify the raw binary files and ship to proper
/pnfs location for tape backup. Need scripting and status to dataProcessingdB.
(9) Swizzling/BeamDaq/MetaData appending and shipping to
/pnfs largely needs integrating into Run Control -- if only via ProcessManager startup/heartbeat monitoring. Need scripting and status to dataProcessingdB.
(10) Need a calibration swizzling module to populate 0th order constants to the http-served calibrationConstants dB (which dB sits offline). This module must compile/link against uboonedaq_datatypes.
(11) A condor system needs to be installed/configured/ up and running on LArTF cluster.) LArSoft + externals needs to be installed on LArTF cluster.
Zarko, David C, Eric(12) Slowmoncon
- VNC server/clients running to view(control) all EPICS CSS panels from anywhere.
- The displays/controls for Weiner MPODs for ASICs power and wire bias, and also for the Weiner PL508s, need to be easily reachable from overview panel.
- The server CPU monitoring, fanspeeds monitored by IPMI needs to be connected to EPICS (either by "real" IOCs or by softIoc fed by a text scraper a la Beam and Cryo)
- The BiRA PMT HV crate needs to be taken and ported over to LArTF (by end July)
- rrd ganglia files need rounding up, displaying by CSS
- EPICS archiving/alarming on ubdaq-prod-smc
- ubdaq-prod-smc failover solution for DAQ databases: runConfig, channelLookup, dataProcessing
(13) Should implement a routine to load patterns into the FPGAs for readout, and then start data validation studies to get measurements of/limits on bit errors.
(14) We need to measure total available data throughput for each of the choke points in the triggered run-mode: time to access DMA, time to copy into circular buffer, time to process/produce fragments from circular buffer, time to place fragments in SEB's outbound queue, time to transfer fragment over ethernet to assembler, time to receive fragments on assembler, time to collect/bundle fragments and form full eventRecord, and time to write eventRecord to disk. Ganglia metrics should be placed at all of these points for proper real-time monitoring.
(15) Need to start proper documentation.
(17) Need to read trigger word over dedicated fiber and merge into data stream. Crate status monitoring working into ganglia metrics, then alarmed by EPICS.
(18) Get 8-port DLink network switches from Andrey Bobishov, mount 1 each to DAQ racks at DAB.
(19) Mount Glomations in racks, with SDRam cards loaded with img file and filesystem.
Glenn, Sowjanya, Andrzej
(20) On-platform electronics install
Bryce, Linda, Eric, Wes, Georgia
(21) The DAQ runConfig dB needs work. Exists, but needs to be exercised
Kazu, Andrzej, Wes, Eric
(22) dataProcessing dB
(23) channelLookup dB needs PMT channel mapping. Needs work generally for consistency.
Jason, David K
(24) The scripting to commandeer the strobe pulser for flasher runs needs doing. This involves configuring the pulser and sending pulses from that pulser to the flasher system on receipt of software triggers -- which the trigger board will provide. Ranger needs to run one USB port from seb10 for flasher box.
(25) Establish SN readout mode. We need first to test this at DAB Annex. We need scripts to monitor, move and reap this huge volume of data.
(26) Run with compression/Huffman-enoding: both triggered and SN modes. This is pending updated SN compression scheme from Nevis.
Eric, Wes, David C
(27) Develop Laser mode. A design is understood, and coding is in process
Wes, Eric, Matthias, Thomas
(28) SN processing: with sensitivity to network bandwidth, we need SN scripts and processes to (a) generate/listen for a signal (fake/real email from SNEWs), upon receipt of which we will (b) end DAQ run, start it up again in relevant simple beam-only mode to (c) awaken processes on sebs to gather 3 hrs of data on disk around signal timestamp and ship to evb (c) awaken SNassembler to receive these fragments, build and write to disk (d) back up raw data to enstore, swizzle and send that also to enstore. (e) add ganglia metrics, online histograms as relevant (f) interface to Run Control .... deep integration here with the data processing dB.
(30) Run Control needs to run Flasher calibration, Laser and triggered data modes.
David K, Anne, Joseph
(31) Tibbs disk back-ups of /data on all servers, and /uboone
(32) Put ubdaq-prod-smc management under SCD-FEF group?
(33) XOC move-in/function