Project

General

Profile

Wiki » History » Version 38

Lu Ren, 08/24/2018 12:12 PM

1 2 Lu Ren
h1. MicroBooNE Production
2 1 Lu Ren
3 14 Lu Ren
h2. Request a Collaboration wide production sample 
4 1 Lu Ren
5 14 Lu Ren
* Figure out what you need for the sample and fill out production request form "[click me]":https://docs.google.com/forms/d/e/1FAIpQLSc0MMmWL5hYFg-tuAT7eUpCvVAIau8hg9hwOfFE149GLJcj0Q/viewform
6 27 Wei Tang
7 26 Wei Tang
* You might need to present your sample request at Analysis Tool Meeting and/or Data Management Meeting
8 27 Wei Tang
9 1 Lu Ren
* The production is ONLY effective with physics conveners' approval. 
10 27 Wei Tang
11 30 Wei Tang
* The fcl files for the production are prepared by user/group who requests the sample and they are also responsible for making sure the production workflow has been tested and is correct.
12 27 Wei Tang
13 1 Lu Ren
* The user/group are also responsible for validating the sample produced by production team that reaches their expectations. 
14 1 Lu Ren
15 27 Wei Tang
16 27 Wei Tang
h2. Coordination for a large data sample pre-staging
17 27 Wei Tang
18 27 Wei Tang
MicroBooNE requires the pre-staging of dataset larger than "dev" sample (4000 files or 10 TB file size) to be coordinated with production team.  
19 27 Wei Tang
20 27 Wei Tang
* Before making pre-staging request, check pre-staging dataset status table if your requested sample has already been pre-staged. How long the pre-staging last for the dataset depends on various factors. A rule of thumb is if the dataset has been pre-staged within a couple of weeks, then most likely you don't have to pre-stage it. 
21 27 Wei Tang
22 27 Wei Tang
* Check the dataset information with samweb commands 
23 27 Wei Tang
   >$ samweb list-files --summary "defname:Your_Dataset_Name"
24 27 Wei Tang
25 1 Lu Ren
26 31 Wei Tang
   Here is an example:
27 31 Wei Tang
   samweb list-files --summary "defname:prodgenie_bnb_intrinsic_nue_cosmic_uboone_mcc8.7_reco2" 
28 31 Wei Tang
   File count:    12287           (file numbers)
29 31 Wei Tang
   Total size:    34235530158207     (file size in byte)
30 31 Wei Tang
   Event count:    614350       (available events)
31 27 Wei Tang
32 5 Lu Ren
h2. Recommended Data and MC Samples for analysis (as of Aug. 20th, 2018)
33 2 Lu Ren
34 12 Lu Ren
35 12 Lu Ren
Data:
36 12 Lu Ren
* "Good Run Selected Data for 2018":https://microboone-exp.fnal.gov/at_work/AnalysisTools/data/ub_data_2018.html 
37 12 Lu Ren
38 12 Lu Ren
MC:
39 1 Lu Ren
* "MCC8.7 Central Value MC":https://microboone-exp.fnal.gov/at_work/AnalysisTools/mc/mcc8.7/details.html
40 1 Lu Ren
* "MCC8.8 NuMI MC":https://microboone-exp.fnal.gov/at_work/AnalysisTools/mc/mcc8.8/ub_mc_numi.html
41 36 Lu Ren
* "Detector variation samples":https://microboone-exp.fnal.gov/at_work/AnalysisTools/mc/mcc8.10/det_syst.html ("data format":https://microboone-docdb.fnal.gov/cgi-bin/private/ShowDocument?docid=16010 and "description":https://microboone-docdb.fnal.gov/cgi-bin/private/ShowDocument?docid=16028)
42 12 Lu Ren
43 9 Lu Ren
h2. Status of Dataset Prestaging
44 8 Lu Ren
45 7 Lu Ren
|_. Dataset |_. # of files|_. Size|_. Status|_. Requested by |_. Approved|_. Notes |
46 2 Lu Ren
| |  |  | | | | |
47 2 Lu Ren
48 1 Lu Ren
h2. Data and MC samples
49 1 Lu Ren
50 12 Lu Ren
Find full list of samples at "MicroBooNE at Work":https://microboone-exp.fnal.gov/at_work/AnalysisTools/index.html
51 6 Lu Ren
52 23 Anna Mazzacane
h2. List of Production Requests 
53 23 Anna Mazzacane
54 23 Anna Mazzacane
Find actual list of production requests to the team at "List of Production Requests":https://docs.google.com/spreadsheets/d/1yr2KmuzlnLtoEWzTxqd_SWW_mKUluFaGUYXWbv4h5wM/edit#gid=1994969007https://docs.google.com/spreadsheets/d/1yr2KmuzlnLtoEWzTxqd_SWW_mKUluFaGUYXWbv4h5wM/edit#gid=1994969007
55 23 Anna Mazzacane
56 6 Lu Ren
h2. For production team members
57 1 Lu Ren
58 1 Lu Ren
[[How-to's for new team members]]
59 1 Lu Ren
[[Offline Production shift]]
60 38 Lu Ren
"Data Blinding":https://cdcvs.fnal.gov/redmine/projects/uboonecode/wiki/Blinding_of_MicroBooNE_Data
61 37 Lu Ren
"How to create a new data tier":https://cdcvs.fnal.gov/redmine/projects/uboonecode/wiki/Creating_a_New_Data_Tier
62 1 Lu Ren
"MCC9 Production Plan":https://cdcvs.fnal.gov/redmine/projects/uboone-physics-analysis/wiki/MCC9_Production_Plan
63 35 Lu Ren
"Pubs Era Production Status":https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki/Production_-_Status
64 1 Lu Ren
65 1 Lu Ren
h2. For users
66 27 Wei Tang
67 32 Wei Tang
*Before submitting large number of jobs to the grid, first make sure you have tested your workflow and it is correct.* Also make sure your resources request setup follows the uB's Grid Best Practices 
68 32 Wei Tang
* "Herb's Grid Best Practice":https://microboone-docdb.fnal.gov/cgi-bin/private/RetrieveFile?docid=14184&filename=uboone_grid_feb27_2018.pdf&version=2
69 32 Wei Tang
* "Kirby and Wei's Best Practice":https://microboone-docdb.fnal.gov/cgi-bin/private/RetrieveFile?docid=16777&filename=uB_grid_tutorial_07_23_2018_v2.pdf&version=2
70 32 Wei Tang
* "Herb's real life example of how to use recursive dataset":https://microboone-docdb.fnal.gov/cgi-bin/private/RetrieveFile?docid=13942&filename=uboone_larbatch_feb15_2018.pdf&version=1
71 34 Wei Tang
* You can also check [[Matt's Summary of grid best practice]]
72 32 Wei Tang
   
73 27 Wei Tang
Please "contact the production team":mailto:microboone_offline_production@listserv.fnal.gov, if you 
74 27 Wei Tang
* Want to prestage and process a dataset with more than 4000 files or size > 10 TB
75 27 Wei Tang
* Cancel large number (>100) of jobs on the grid
76 27 Wei Tang
* Have questions about running jobs on the grid