Project

General

Profile

Wiki » History » Version 110

Version 109 (Lu Ren, 05/07/2019 01:46 PM) → Version 110/127 (Lu Ren, 05/07/2019 01:46 PM)

h1. MicroBooNE Production

h2. {{collapse(Request a Collaboration wide production sample)

* Figure out what you need for the sample and fill out production request form "[click me]":https://docs.google.com/forms/d/e/1FAIpQLSc0MMmWL5hYFg-tuAT7eUpCvVAIau8hg9hwOfFE149GLJcj0Q/viewform

* You might need to present your sample request at Analysis Tool Meeting and/or Data Management Meeting

* The production is ONLY effective with physics conveners' approval.

* The fcl files for the production are prepared by user/group who requests the sample and they are also responsible for making sure the production workflow has been tested and is correct.

* The user/group are also responsible for validating the sample produced by production team that reaches their expectations.

}}

h2. Coordination for a large data sample pre-staging

MicroBooNE requires the pre-staging of dataset larger than "dev" sample (4000 files or 10 TB file size) to be coordinated with production team.

* Before making pre-staging request, check pre-staging dataset status table if your requested sample has already been pre-staged. How long the pre-staging last for the dataset depends on various factors. A rule of thumb is if the dataset has been pre-staged within a couple of weeks, then most likely you don't have to pre-stage it.

* Check the dataset information with samweb commands ("learn more about SAM":https://cdcvs.fnal.gov/redmine/projects/uboonecode/wiki/Sam) ("More SAM commands":https://cdcvs.fnal.gov/redmine/projects/sam-main/wiki/Sam_web_client_Command_Reference)
<pre>
$ samweb list-files --summary "defname:Your_Dataset_Name"
</pre>

Here is an example:
<pre>
samweb list-files --summary "defname:prodgenie_bnb_intrinsic_nue_cosmic_uboone_mcc8.7_reco2"
File count:    12287 (file numbers)
Total size:    34235530158207 (file size in byte)
Event count:    614350 (available events)
</pre>

h2. Recommended Data and MC Samples for MCC8 analysis (as of Aug. 20th 2018)

Data:
* "Good Run Selected Data for 2018":https://microboone-exp.fnal.gov/at_work/AnalysisTools/data/ub_data_2018.html

MC:
* "MCC8.7 Central Value MC":https://microboone-exp.fnal.gov/at_work/AnalysisTools/mc/mcc8.7/details.html
* "MCC8.8 NuMI MC":https://microboone-exp.fnal.gov/at_work/AnalysisTools/mc/mcc8.8/ub_mc_numi.html
* "Detector variation samples I":https://microboone-exp.fnal.gov/at_work/AnalysisTools/mc/mcc8.10/det_syst.html ("data format":https://microboone-docdb.fnal.gov/cgi-bin/private/ShowDocument?docid=16010 and "description":https://microboone-docdb.fnal.gov/cgi-bin/private/ShowDocument?docid=16028)
* "Detector variation samples II":https://microboone-exp.fnal.gov/at_work/AnalysisTools/mc/mcc8.17/det_syst.html

h2. Status of Dataset Prestaging

|_. Dataset |_. # of files|_. Size|_. Requested by |_. Date|_. Status|
|prodgenie_numi_cosmic_uboone_mcc8.8_detsim_mec_reco2|7.3k|37 TB|Krishan Mistry|04/04/2019|active|
|prodgenie_bnb_nu_cosmic_uboone_MCC8Tune3_mcc8.7_reco2_mcc8.23_ccinfilter |||Kirby|04/15/2019|active|
|prodgenie_bnb_nu_uboone_overlay_mcc8.11_reco2|18.5k|16.3 TB|Afroditi Papadopoulou|04/23/2019|active|
|janthony_prodgenie_bnb_nu_cosmic_uboone_mcc8.7_reco2_batch_4_of_4~|10k||Lorena|05/07/2019|active| |janthony_prodgenie_bnb_nu_cosmic_uboone_mcc8.7_reco2_batch_4_of_4~|10k|Lorena|05/07/2019|active|

h2. Data and MC samples

Find full list of samples at "MicroBooNE at Work":https://microboone-exp.fnal.gov/at_work/AnalysisTools/index.html

h2. List of Production Requests

Find actual list of production requests to the team at "List of Production Requests":https://docs.google.com/spreadsheets/d/1yr2KmuzlnLtoEWzTxqd_SWW_mKUluFaGUYXWbv4h5wM/edit#gid=1994969007https://docs.google.com/spreadsheets/d/1yr2KmuzlnLtoEWzTxqd_SWW_mKUluFaGUYXWbv4h5wM/edit#gid=1994969007

h2. For production team members

[[How-to's and useful information]]
[[Meetings and Minutes]]
[[Offline Production Shift]]
"MCC9 Production":https://cdcvs.fnal.gov/redmine/projects/uboone-physics-analysis/wiki/MCC9_Production_Plan
[[Detector variation sample production]]
[[Optical Filter Production]]
[[Archiving Inactive Users]]
"Data Blinding":https://cdcvs.fnal.gov/redmine/projects/uboonecode/wiki/Blinding_of_MicroBooNE_Data
"Pubs Era Production Status":https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki/Production_-_Status

h2. For users

*Before submitting large number of jobs to the grid, first make sure you have tested your workflow and it is correct.* Also make sure your resources request setup follows the uB's Grid Best Practices
* "Herb's Grid Best Practice":https://microboone-docdb.fnal.gov/cgi-bin/private/RetrieveFile?docid=14184&filename=uboone_grid_feb27_2018.pdf&version=2
* "Kirby and Wei's Best Practice":https://microboone-docdb.fnal.gov/cgi-bin/private/RetrieveFile?docid=16777&filename=uB_grid_tutorial_07_23_2018_v2.pdf&version=2
* "Herb's real life example of how to use recursive dataset":https://microboone-docdb.fnal.gov/cgi-bin/private/RetrieveFile?docid=13942&filename=uboone_larbatch_feb15_2018.pdf&version=1
* You can also check [[Matt's Summary of grid best practice]]

Please "contact the production team":mailto:microboone_offline_production@listserv.fnal.gov, if you
* Want to prestage and process a dataset with more than 4000 files or size > 10 TB
* Cancel large number (>100) of jobs on the grid
* Have questions about running jobs on the grid

h2. Collection of legacy wiki pages

"Predefined Datasets Dec. 2015":https://cdcvs.fnal.gov/redmine/projects/uboonecode/wiki/Predefined_Datasets