Project

General

Profile

Wiki » History » Version 91

Lu Ren, 02/19/2019 02:40 PM

1 2 Lu Ren
h1. MicroBooNE Production
2 1 Lu Ren
3 48 Lu Ren
h2. {{collapse(Request a Collaboration wide production sample)
4 1 Lu Ren
5 14 Lu Ren
* Figure out what you need for the sample and fill out production request form "[click me]":https://docs.google.com/forms/d/e/1FAIpQLSc0MMmWL5hYFg-tuAT7eUpCvVAIau8hg9hwOfFE149GLJcj0Q/viewform
6 27 Wei Tang
7 26 Wei Tang
* You might need to present your sample request at Analysis Tool Meeting and/or Data Management Meeting
8 27 Wei Tang
9 1 Lu Ren
* The production is ONLY effective with physics conveners' approval. 
10 27 Wei Tang
11 30 Wei Tang
* The fcl files for the production are prepared by user/group who requests the sample and they are also responsible for making sure the production workflow has been tested and is correct.
12 27 Wei Tang
13 42 Lu Ren
* The user/group are also responsible for validating the sample produced by production team that reaches their expectations.
14 50 Lu Ren
15 48 Lu Ren
}}
16 27 Wei Tang
17 27 Wei Tang
h2. Coordination for a large data sample pre-staging
18 27 Wei Tang
19 27 Wei Tang
MicroBooNE requires the pre-staging of dataset larger than "dev" sample (4000 files or 10 TB file size) to be coordinated with production team.  
20 27 Wei Tang
21 27 Wei Tang
* Before making pre-staging request, check pre-staging dataset status table if your requested sample has already been pre-staged. How long the pre-staging last for the dataset depends on various factors. A rule of thumb is if the dataset has been pre-staged within a couple of weeks, then most likely you don't have to pre-stage it. 
22 1 Lu Ren
23 46 Wei Tang
* Check the dataset information with samweb commands  ("learn more about SAM":https://cdcvs.fnal.gov/redmine/projects/uboonecode/wiki/Sam) ("More SAM commands":https://cdcvs.fnal.gov/redmine/projects/sam-main/wiki/Sam_web_client_Command_Reference)
24 42 Lu Ren
<pre>
25 42 Lu Ren
$ samweb list-files --summary "defname:Your_Dataset_Name"
26 42 Lu Ren
</pre>
27 42 Lu Ren
   
28 1 Lu Ren
   Here is an example:
29 42 Lu Ren
<pre>
30 42 Lu Ren
  samweb list-files --summary "defname:prodgenie_bnb_intrinsic_nue_cosmic_uboone_mcc8.7_reco2" 
31 31 Wei Tang
   File count:    12287           (file numbers)
32 31 Wei Tang
   Total size:    34235530158207     (file size in byte)
33 42 Lu Ren
   Event count:    614350       (available events)
34 43 Lu Ren
</pre>
35 27 Wei Tang
36 66 Lu Ren
h2. Recommended Data and MC Samples for MCC8 analysis (as of Aug. 20th 2018)
37 2 Lu Ren
38 12 Lu Ren
39 12 Lu Ren
Data:
40 12 Lu Ren
* "Good Run Selected Data for 2018":https://microboone-exp.fnal.gov/at_work/AnalysisTools/data/ub_data_2018.html 
41 12 Lu Ren
42 12 Lu Ren
MC:
43 1 Lu Ren
* "MCC8.7 Central Value MC":https://microboone-exp.fnal.gov/at_work/AnalysisTools/mc/mcc8.7/details.html
44 1 Lu Ren
* "MCC8.8 NuMI MC":https://microboone-exp.fnal.gov/at_work/AnalysisTools/mc/mcc8.8/ub_mc_numi.html
45 66 Lu Ren
* "Detector variation samples I":https://microboone-exp.fnal.gov/at_work/AnalysisTools/mc/mcc8.10/det_syst.html ("data format":https://microboone-docdb.fnal.gov/cgi-bin/private/ShowDocument?docid=16010 and "description":https://microboone-docdb.fnal.gov/cgi-bin/private/ShowDocument?docid=16028)
46 66 Lu Ren
* "Detector variation samples II":https://microboone-exp.fnal.gov/at_work/AnalysisTools/mc/mcc8.17/det_syst.html
47 67 Lu Ren
48 9 Lu Ren
h2. Status of Dataset Prestaging
49 1 Lu Ren
50 73 Wei Tang
|_. Dataset |_. # of files|_. Size|_. Requested by |_. Date|_. Status|
51 63 Lu Ren
|prod_reco_optfilter_extbnb_v12_mcc8_gooodruns_v4s
52 78 Wei Tang
prod_reco_optfilter_extbnb_v12_mcc8_gooodruns_v5s|||single photon group (Mark, Kathryn)||expire|
53 78 Wei Tang
|prod_numi_ext_data_run1_reco2_mcc8_v1_goodruns |16209|56TB|Colton||expire|
54 1 Lu Ren
|prodgenie_bnb_nu_cosmic_uboone_mcc8.7_reco2
55 78 Wei Tang
prodgenie_bnb_nu_cosmic_uboone_MCC8Tune3_mcc8.7_reco2|||Marco||expire|
56 78 Wei Tang
|prod_numi_data_run1_reco2_mcc8_v1_no_anumi_goodruns|25748|~62TB|Nicolo|11/30/2018|expire|
57 78 Wei Tang
|prod_numi_ext_data_run1_reco2_mcc8_v1_goodruns|16209|~51TB|Nicolo|11/30/2018|expire|
58 78 Wei Tang
|prodgenie_numi_cosmic_uboone_mcc8.8_detsim_mec_reco2|7322|~37TB|Nicolo|11/30/2018|expire|
59 78 Wei Tang
|prodgenie_bnb_intrinsic_nue_cosmic_uboone_mcc8.7_reco2|12210|~31TB|Nicolo|11/30/2018|expire|
60 85 Wei Tang
|mcc8.17 detector variation samples|4k*19|~|Kirby|12/27/2018|expire|
61 87 Wei Tang
|NuMI EXT||2.5TB|cbarnes4|1/7/2019|expire|
62 87 Wei Tang
|NuMI |||chill2|1/7/2019|expire|
63 81 Wei Tang
|prod_bnbhsnc0_swizzle_crt_inclusive_mixv5v6_R3_r13696to15500|15k|6.5 TB|Salvatore Davide Porzio|01/16/2019|active|
64 81 Wei Tang
|prod_bnbhsnc0_swizzle_crt_inclusive_mixv5v6_R3_r15500to17055|14k|3.7 TB|Salvatore Davide Porzio|01/16/2019|active|
65 81 Wei Tang
|prod_exthsnc0_swizzle_crt_inclusive_v6_R3_r13696to15500|12k|1.5 TB|Salvatore Davide Porzio|01/16/2019|active|
66 81 Wei Tang
|prod_exthsnc0_swizzle_crt_inclusive_v6_R3_r15500to17055|14k|2.0 TB|Salvatore Davide Porzio|01/16/2019|active|
67 83 Wei Tang
|prodgenie_bnb_nu_cosmic_uboone_mcc8.7_reco2_dev|4k|10 TB|Afroditi Papadopoulou|01/16/2019|active|
68 83 Wei Tang
|prod_reco_optfilter_bnb_v14_unblind_mcc8|4.1k|3.23 TB|Adam Lister|01/17/2019|active|
69 83 Wei Tang
|prod_reco_optfilter_extbnb_ccinclusive_v15_mcc8|10.6k|0.35 TB|Adam Lister|01/17/2019|active|
70 83 Wei Tang
|prodgenie_bnb_nu_cosmic_uboone_mcc8.7_reco2|43.5k|110 TB|Adam Lister|01/17/2019|active|
71 84 Lu Ren
|prod_numi_ext_data_run1_reco2_mcc8_v1 |||cbarnes4|1/24/2019||
72 1 Lu Ren
|prodgenie_numi_cosmic_uboone_mcc8.8_detsim_mec_reco2|||cbarnes4|1/24/2019||
73 86 Wei Tang
|prod_reco_optfilter_bnb_ccinclusive_v14_mcc8|14.3k|0.5 TB|Varuna Meddage|01/25/2019|active|
74 83 Wei Tang
|prodgenie_numi_cosmic_uboone_mcc8.8_detsim_mec_reco2|7.3k|37 TB|Nicolò Foppiani|02/02/2019|active|
75 89 Wei Tang
|Various datasets for SPECalibration|~25k in total| |Sharankova, Ralitsa|02/19/2019|active|
76 91 Lu Ren
|mcc8 run 1 extbnb|||amogan,alister1|02/19/2019/active|
77 2 Lu Ren
78 1 Lu Ren
h2. Data and MC samples
79 1 Lu Ren
80 12 Lu Ren
Find full list of samples at "MicroBooNE at Work":https://microboone-exp.fnal.gov/at_work/AnalysisTools/index.html
81 6 Lu Ren
82 23 Anna Mazzacane
h2. List of Production Requests 
83 23 Anna Mazzacane
84 23 Anna Mazzacane
Find actual list of production requests to the team at "List of Production Requests":https://docs.google.com/spreadsheets/d/1yr2KmuzlnLtoEWzTxqd_SWW_mKUluFaGUYXWbv4h5wM/edit#gid=1994969007https://docs.google.com/spreadsheets/d/1yr2KmuzlnLtoEWzTxqd_SWW_mKUluFaGUYXWbv4h5wM/edit#gid=1994969007
85 23 Anna Mazzacane
86 6 Lu Ren
h2. For production team members
87 1 Lu Ren
88 69 Lu Ren
[[How-to's and useful information]]
89 40 Lu Ren
[[Meetings and Minutes]]
90 1 Lu Ren
[[Offline Production Shift]]
91 68 Lu Ren
"MCC9 Production":https://cdcvs.fnal.gov/redmine/projects/uboone-physics-analysis/wiki/MCC9_Production_Plan
92 61 Lu Ren
[[Detector variation sample production]]
93 65 Lu Ren
[[Optical Filter Production]]
94 88 Lu Ren
[[Archiving Inactive Users]]
95 60 Lu Ren
"Data Blinding":https://cdcvs.fnal.gov/redmine/projects/uboonecode/wiki/Blinding_of_MicroBooNE_Data
96 35 Lu Ren
"Pubs Era Production Status":https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki/Production_-_Status
97 1 Lu Ren
98 1 Lu Ren
h2. For users
99 27 Wei Tang
100 32 Wei Tang
*Before submitting large number of jobs to the grid, first make sure you have tested your workflow and it is correct.* Also make sure your resources request setup follows the uB's Grid Best Practices 
101 32 Wei Tang
* "Herb's Grid Best Practice":https://microboone-docdb.fnal.gov/cgi-bin/private/RetrieveFile?docid=14184&filename=uboone_grid_feb27_2018.pdf&version=2
102 32 Wei Tang
* "Kirby and Wei's Best Practice":https://microboone-docdb.fnal.gov/cgi-bin/private/RetrieveFile?docid=16777&filename=uB_grid_tutorial_07_23_2018_v2.pdf&version=2
103 32 Wei Tang
* "Herb's real life example of how to use recursive dataset":https://microboone-docdb.fnal.gov/cgi-bin/private/RetrieveFile?docid=13942&filename=uboone_larbatch_feb15_2018.pdf&version=1
104 34 Wei Tang
* You can also check [[Matt's Summary of grid best practice]]
105 32 Wei Tang
   
106 27 Wei Tang
Please "contact the production team":mailto:microboone_offline_production@listserv.fnal.gov, if you 
107 27 Wei Tang
* Want to prestage and process a dataset with more than 4000 files or size > 10 TB
108 27 Wei Tang
* Cancel large number (>100) of jobs on the grid
109 27 Wei Tang
* Have questions about running jobs on the grid
110 47 Lu Ren
111 58 Lu Ren
h2. Collection of legacy wiki pages
112 58 Lu Ren
113 57 Lu Ren
"Predefined Datasets Dec. 2015":https://cdcvs.fnal.gov/redmine/projects/uboonecode/wiki/Predefined_Datasets