Lu Ren, 08/28/2018 10:19 AM
h1. MicroBooNE Production
h2. Request a Collaboration wide production sample
* Figure out what you need for the sample and fill out production request form "[click me]":https://docs.google.com/forms/d/e/1FAIpQLSc0MMmWL5hYFg-tuAT7eUpCvVAIau8hg9hwOfFE149GLJcj0Q/viewform
* You might need to present your sample request at Analysis Tool Meeting and/or Data Management Meeting
* The production is ONLY effective with physics conveners' approval.
* The fcl files for the production are prepared by user/group who requests the sample and they are also responsible for making sure the production workflow has been tested and is correct.
* The user/group are also responsible for validating the sample produced by production team that reaches their expectations.
h2. Coordination for a large data sample pre-staging
MicroBooNE requires the pre-staging of dataset larger than "dev" sample (4000 files or 10 TB file size) to be coordinated with production team.
* Before making pre-staging request, check pre-staging dataset status table if your requested sample has already been pre-staged. How long the pre-staging last for the dataset depends on various factors. A rule of thumb is if the dataset has been pre-staged within a couple of weeks, then most likely you don't have to pre-stage it.
* Check the dataset information with samweb commands
$ samweb list-files --summary "defname:Your_Dataset_Name"
Here is an example:
samweb list-files --summary "defname:prodgenie_bnb_intrinsic_nue_cosmic_uboone_mcc8.7_reco2"
File count: 12287 (file numbers)
Total size: 34235530158207 (file size in byte)
Event count: 614350 (available events)
h2. Recommended Data and MC Samples for analysis (as of Aug. 20th, 2018)
* "Good Run Selected Data for 2018":https://microboone-exp.fnal.gov/at_work/AnalysisTools/data/ub_data_2018.html
* "MCC8.7 Central Value MC":https://microboone-exp.fnal.gov/at_work/AnalysisTools/mc/mcc8.7/details.html
* "MCC8.8 NuMI MC":https://microboone-exp.fnal.gov/at_work/AnalysisTools/mc/mcc8.8/ub_mc_numi.html
* "Detector variation samples":https://microboone-exp.fnal.gov/at_work/AnalysisTools/mc/mcc8.10/det_syst.html ("data format":https://microboone-docdb.fnal.gov/cgi-bin/private/ShowDocument?docid=16010 and "description":https://microboone-docdb.fnal.gov/cgi-bin/private/ShowDocument?docid=16028)
h2. Status of Dataset Prestaging
|_. Dataset |_. # of files|_. Size|_. Status|_. Requested by |_. Approved|_. Notes |
| | | | | | | |
h2. Data and MC samples
Find full list of samples at "MicroBooNE at Work":https://microboone-exp.fnal.gov/at_work/AnalysisTools/index.html
h2. List of Production Requests
Find actual list of production requests to the team at "List of Production Requests":https://docs.google.com/spreadsheets/d/1yr2KmuzlnLtoEWzTxqd_SWW_mKUluFaGUYXWbv4h5wM/edit#gid=1994969007https://docs.google.com/spreadsheets/d/1yr2KmuzlnLtoEWzTxqd_SWW_mKUluFaGUYXWbv4h5wM/edit#gid=1994969007
h2. For production team members
[[How-to's for new team members]]
[[Meetings and Minutes]]
[[Offline Production Shift]]
"MCC9 Production Plan":https://cdcvs.fnal.gov/redmine/projects/uboone-physics-analysis/wiki/MCC9_Production_Plan
"Pubs Era Production Status":https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki/Production_-_Status
h2. For users
*Before submitting large number of jobs to the grid, first make sure you have tested your workflow and it is correct.* Also make sure your resources request setup follows the uB's Grid Best Practices
* "Herb's Grid Best Practice":https://microboone-docdb.fnal.gov/cgi-bin/private/RetrieveFile?docid=14184&filename=uboone_grid_feb27_2018.pdf&version=2
* "Kirby and Wei's Best Practice":https://microboone-docdb.fnal.gov/cgi-bin/private/RetrieveFile?docid=16777&filename=uB_grid_tutorial_07_23_2018_v2.pdf&version=2
* "Herb's real life example of how to use recursive dataset":https://microboone-docdb.fnal.gov/cgi-bin/private/RetrieveFile?docid=13942&filename=uboone_larbatch_feb15_2018.pdf&version=1
* You can also check [[Matt's Summary of grid best practice]]
Please "contact the production team":mailto:email@example.com, if you
* Want to prestage and process a dataset with more than 4000 files or size > 10 TB
* Cancel large number (>100) of jobs on the grid
* Have questions about running jobs on the grid