Project

General

Profile

Minos Beam processing

This "Beam Data" is a bit more general than simply that of the batch group efforts, but lives here for historical reasons as well as a lack of an obvious alternative.

The topics to be discussed here include:

  • acquisition of the data from ACNet into MINOS .mbeam.root format files
  • processing said .mbeam.root files into MINOS DB BeamMonSpill table entries
  • fixing or repopulating BeamMonSpill when there are incomplete .mbeam.root files
  • verifying that the MINOS DB is up-to-date before batch processing of files proceeds
  • reworking Jas online beam monitoring in the MINOS+ era
  • this is not a batch task, but is only put here because the effort shares many of the same concerns with the .mbeam.root upgrade task

Contacts

Documents

Minos Beam logging Status Summary

Beam Database missing beam summary (fixup periods) BEAMDB

NuMI Device Summary ( Devices, Descriptions )

NuMI Device Summary

Beam File Processing overview

  • cf2root - mindata@minos27
    • Data from the collector files are merged to produce traditional MINOS *.mbeam.root files
    • Files are written to /minos/app/BEAMDATA_CF/MBEAM
    • Empty pointer files are placed in the /minos/app/BEAMDATA_CF/MBEAM/archiver/data-to-archive subdirectory for archiver
    • mindata@minos27.fnal.gov should have crontab -l entry:
        MAILTO="rhatcher@fnal.gov" 
        # run script to catalogue beam 'collector files' 30 min after the hour every hour
        # script will also make .mbeam.root files at 00 08 16 hour UTC
        30 * * * * /minos/app/home/mindata/bdcfcron
      
    • Instructions on (re)generating individual files can be found in this section
  • archiver - mindata@minos-nearline
    • Files are copied via Kerberized FTP to /pnfs/minos/beam_data/YYYY-MM/...
    • Pointers are moved to ../data-archived
    • Normally handled by a crontab line in CVS file admin/crontab/minos-nearline.mindata
      @reboot sleep 600 ; ${HOME}/bin/init archiver start
      
    • to start manually, log into mindata@minos-nearline.fnal.gov and
        {HOME}/bin/init archiver start
        {HOME}/bin/init archiver status
      
    • to stop {HOME}/bin/init archiver stop
  • beamdbu - mindata@minos27
    • Files are processed by two loon scripts, results are written to the offline database.
    • Pointers are moved to ../data-dbued.
    • Logs for each file are in /minos/app/BEAMDATA_CF/MBEAM/archiver/LOG
        tail -f /minos/app/BEAMDATA_CF/MBEAM/archiver/LOG/beamdbu.log
      
    • CVS crontab/minos27.mindata contains this line to start after reboot
      @reboot sleep 600 ; ${HOME}/beamdbu > /dev/null 2>&1
      
    • MANUAL START : log into mindata@minos27.fnal.gov
        set nohup ; ${HOME}/beamdbu &
      
    • MANUAL STOP: also as mindata@minos27.fnal.gov
        touch /minos/app/BEAMDATA_CF/MBEAM/archiver/LOG/STOP
      

Information about IFBeamData DB

There are separate historical and realtime databases.

  • Historical
    • Retention policies are
      • long - forever - including bundle NuMI_Physics
      • short - one month - including bundle NuMI_Monitoring
  • Realtime
    • All devices are available
    • Usually available within a second or so of real time
    • Rention is one hour
Readouts by bundle name access only the Historical database.
Readouts by device name access both Historical and Realtime databased.
  • Use curl -L to redirect to the backend web servers.
  • Note the typical 9 second time gap after 1334332808678
  • For NuMI running starting Aug 2013, the readout event is e,a9,e,500
    $ curl -L 'http://ifb-data.fnal.gov:8089/ifbeam/data/data?v=E:TR101D&e=e,a9&t0=1334332800&t1=1334332830&f=csv'
    Event,Variable,Clock,Units,Value(s)
    "e,a9",E:TR101D,1334332800409,E12,37.24438
    "e,a9",E:TR101D,1334332802476,E12,37.10143
    "e,a9",E:TR101D,1334332804543,E12,37.075905
    "e,a9",E:TR101D,1334332806611,E12,37.332874
    "e,a9",E:TR101D,1334332808678,E12,37.203537
    "e,a9",E:TR101D,1334332817281,E12,35.597042
    "e,a9",E:TR101D,1334332819349,E12,37.150784
    "e,a9",E:TR101D,1334332821416,E12,37.02655
    "e,a9",E:TR101D,1334332823484,E12,37.198433
    "e,a9",E:TR101D,1334332825551,E12,37.021446
    "e,a9",E:TR101D,1334332827618,E12,37.05548
    "e,a9",E:TR101D,1334332829685,E12,37.307346
    
    $ curl -L 'http://ifb-data.fnal.gov:8089/ifbeam/data/data?v=E:TR101D&e=e,a9,e,500&t0=1377903600&t1=1377903700&f=csv'
    Event,Variable,Clock,Units,Value(s)
    "e,a9,e,500",E:TR101D,1377903604381,E12,0.684691828064
    "e,a9,e,500",E:TR101D,1377903614379,E12,0.684691828064
    "e,a9,e,500",E:TR101D,1377903624378,E12,0.677884642937
    "e,a9,e,500",E:TR101D,1377903634377,E12,0.691499013192
    "e,a9,e,500",E:TR101D,1377903644376,E12,0.682990031783
    "e,a9,e,500",E:TR101D,1377903654375,E12,0.677884642937
    "e,a9,e,500",E:TR101D,1377903664373,E12,0.682990031783
    "e,a9,e,500",E:TR101D,1377903674372,E12,0.684691828064
    "e,a9,e,500",E:TR101D,1377903684372,E12,0.68979721691
    "e,a9,e,500",E:TR101D,1377903694372,E12,0.67107745781
    
    
  • Sample data access for a nice reference period, April 13 2012, 11:00 UTC for 30 seconds.

Beam Data files

Creating individual Beam Data files

The MINOS offline beam data files are standard MINOS offline framework files in ROOT format containg raw data records.

Normally files are created by a crontab job (bdcfcron) run under the mindata account on minos27. If one ever needs to [re]create a file by hand one can use the following procedure:

  MINOS_SETUP_DIR_GRID=/grid/fermiapp/minos/minossoft/setup
  source $MINOS_SETUP_DIR_GRID/setup_minossoft_FNALU.sh -r R3.01 -O
  cf2root [--remake] B131017_160000 
where this is for a file for 2013-10-17 16:00:00 with the default duration of 8 hours.

If a re-created file is already declared to SAM then one might need to follow the procedures found REMOVEBEAMFILE

Inspecting the contents of .mbeam.root files

The subpage linked in the header to this section describes how one might inspect the .mbeam.root files.

Comparison of data from minos root file and from IF database

The subpage linked in the header to this section documents efforts to compare the two data sources.

Filling the MINOS DB BeamMonSpill in the MINOS+ erra

In order to avoid having the rework the code that does calibrations and beam profile fitting the "original" upgrade plan was to create future .mbeam.root files from IF Beam Data DB. This would mean simply replacing the first step that used to use the XMLRPC-interface to ACNet. Alas, that turned out to be impractical (IFBeamDB couldn't support it at the time).

Instead MINOS+ turned to drinking from the fire hose by accepting copies of the "collector files" generated to populate the IFBeamDB. These "collector files" are (very) small compressed text files. Because the files are so small they are directed towards /minos/app/BEAMDATA_CF/IMPORT/collectors/long/ because /minos/data has much larger block sizes that would make putting them there extremely wasteful of disk space.

collector file process - db group
collector file process - minos

The files were directed to MINOS+ as a complete mishmash of all possible collector files, with from all sources with no guarantees about file naming. The first step was to sort them via a process catalogue_cf run as a script under cron every 10 minutes (because MINOS+ get all streams, separate processing handles files that aren't immediately important to MINOS+). These are under /minos/app/BEAMDATA_CF/ARCHIVE.

In addition there is a monitoring script that checks that approximately the right number of files have arrived, and that they generally have the right contents. An archive of that scripts results can be found here: https://nusoft.fnal.gov/minos/beamdata/count_cf.html

Every four hours the bdcfcron process initiates the process cf2root. The cf2root script starts a pair of processes cf2rr and rotorooter; the first reads, sorts, merges collector file data for the given period (adjusting for collector files not tagging data by a consistent spill time), formats it into standard MINOS offline raw data records and sends it via a socket to the standard rotorooter process that writes the ROOT file. The rotorooter is also used by DAQ and DCS to write those files.

In the following picture, the blue octagon was the initial approach to the loss of XMLRPC, but what is currently running is the yellow octagon.

new mbeam_filling diagram

crontab on minos27

 run script to catalogue beam 'collector files' 
# and make .mbeam.root files at 00 08 16 hour UTC
# now-three-subprocesses
# commented out the following three line for 2015 summer shutdown: 
#   need to re-enable them in late september 2015
# re-enabled on 2015-10-02 16:55
# count more often since we're cataloguing more often
# but lower the expectation rate from 48 to 8=(48/6)
#  "--" so that --expect flag is passed to count and not handled by bdcfcron
#
MAILTO="minos_batch@fnal.gov" 
01,11,21,31,41,51 0,1,2,3,4,5,6,7,8,10,11,12,13,14,15,16,17,18,19,20,21,22,23 * * * /minos/app/home/mindata/bdcfcron --quota-check=false -- count --expect=6
11,21,31,41,51 9 * * * /minos/app/home/mindata/bdcfcron --quota-check=false -- count --expect=6
01 9 * * * /minos/app/home/mindata/bdcfcron --quota-check=true -- count --expect=6
04,14,24,34,44,54 * * * * /minos/app/home/mindata/bdcfcron --mv-irrelevant=true catalogue
50                * * * * /minos/app/home/mindata/bdcfcron process

not quite automated Beam Monitoring

Data from queries to the MINOS+ offline database are plotted in 15 min intervals here: https://nusoft.fnal.gov/minos/beammonspill/

Alas, this isn't yet quite automated.... and probably needs migration to a different web area.
Should be running script as mindata in /nusoft/app/web/htdoc/minos/beammonspill

#
MAILTO="rhatcher@fnal.gov" 
# every hour 5min after the hour
5 * * * * /nusoft/app/web/htdoc/minos/beammonspill/make_last12hr_plots > /dev/null

# end-of-crontab

Jas Beam Monitoring

The subpage linked in the header to this section describes efforts to migrate Jas from XMLRPC to the IFBeamData DB

Other