- Table of contents
- DAQ
- Transporter script
- FTS (File Transfer System)
- Finding and getting data
DAQ¶
The DAQ runs on lariat-daq00.
When a run begins, the DAQ configuration is written as an XML file to /daqdata/
.
/daqdata/
:
.time
containing the time the spill began.root
containing the data recorded.complete
indicating the data file is written and closed.
Transporter script¶
The group account lariatraw
runs a cron job of the scripts in ~lariatraw/lariat-online/Xporter/
.
python /home/nfs/lariatraw/lariat-online/Xporter/Xporter.py /daqdata /daqdata/dropbox prod >> /daqdata/Xporter.log
- For new data files
lariat_rXXXXXXX_srYYYY.root
with matching fileslariat_rXXXXXXX_srYYYY.root.time
andlariat_rXXXXXXX_srYYYY.root.complete
, Xporter will try to extract a valid run number and spill number. - If successful, it updates the configuration database from the run's configuration
.xml
file. - Then Xporter creates the SAM metadata
.JSON
file mostly from the ACNET data supplied by the IFBeam database at the time in the.time
file. - Finally Xporter moves the data file and the new
lariat_rXXXXXXX_srYYYY.root.json
file to/daqdata/dropbox/
.
DAQ Configuration Database¶
The databaselariat_prd
listens on port 5443 of ifdb05.fnal.gov.
- The XML parameters of the DAQ configuration are stored for each run in the table
lariat_xml_database
. - The ACNET parameters of each sub-run are read by IFBEAM at mid-spill and end-of-spill, and are stored in the table
lariat_ifbeam_database
.
FTS (File Transfer System)¶
FTS is a Fermilab product. Lariat runs an instance on lariat-daq00, in /home/nfs/lariatraw/FTS/lariatrawFTS/
.
http://lariat-daq00.fnal.gov:8787/fts/status NB: To see this page, log in to lariat-gateway.fnal.gov
and run firefox
, which works from the command line.
lariatrawFTS_fts_config.ini
defines the behavior, which is set as follows:
- Scan for new files matching
/daqdata/dropbox/lariat_r*_sr*.root
every 60 seconds. - If they have a matching file with the same name plus the extra extension,
.json
, attempt to read this as a metadata file for the LAriaT SAM database (see below). - For files with valid SAM metadata, register them with SAM and attempt to move them to tape-backed storage:
enstore:/pnfs/lariat/raw/${file_id/1000000[3]}/${file_id/1000[=3]}
. The file_id is an integer incremented for each file when it becomes an entry in the SAM db. - For successfully transferred files, delete from the /dropbox/ after 1.5 days.
Starting and stopping FTS¶
This will cause FTS to pick up any changes to the configuration file.
cd ~lariatraw/FTS/lariatrawFTS source setup_fts.sh stop_fts $PWD start_fts $PWD $PWD/lariatrawFTS_fts_config.ini
Following a re-start, FTS will take a few minutes to re-establish its previous history of file handling.
Finding and getting data¶
DCache, Enstore, and the SAM Database¶
DCache on /pnfs/
for raw data is in /pnfs/lariat/raw/
. These directories should be readable for any member of the lariat
group. Locating spill files by their metadata (run_number, which systems were on, date of spill, etc) is done through the Lariat SAM database.
On the command line to find the raw data files in the archive, execute the following:
source /grid/fermiapp/products/common/etc/setups.sh setup sam_web_client export SAM_EXPERIMENT=lariat
and then try
samweb --help-commands
Very useful:
samweb list-files --help-dimensions samweb list-parameters secondary.polarity ##Only works for native SAM parameters, not the full list of dimensions. (Some dimensions are SAM parameters, but not most.) samweb list-files "run_number > XXXX"
With the file names you get, you can then do
samweb locate-file lariat_rXXXXXXX_srYYYY.root
The output contains a (bit you can ignore@which tape something), but also the file location. SAM lets you create dataset definitions using the values of the metadata:
- Pre-made datasets for LAriAT
- SAM syntax
- Troubleshooting page for all batch submission and data processing.
Special note about 2014 Beam Line Run Data Files¶
The 2014 beam line runs in August and early September did not have the SAM database
mechanism in place. Also, they are exclusively in the "Classic DAQ" format ending
in .dat extension. They have been archived at the area:
/pnfs/lariat/raw/2014/
If you wish to convert them to the ArtDaq style .root file, refer to the example conversion
script in the lariat-online package here: