Project

General

Profile

Wiki » History » Version 28

Wei Tang, 08/21/2018 10:41 PM

1 2 Lu Ren
h1. MicroBooNE Production
2 1 Lu Ren
3 14 Lu Ren
h2. Request a Collaboration wide production sample 
4 1 Lu Ren
5 14 Lu Ren
* Figure out what you need for the sample and fill out production request form "[click me]":https://docs.google.com/forms/d/e/1FAIpQLSc0MMmWL5hYFg-tuAT7eUpCvVAIau8hg9hwOfFE149GLJcj0Q/viewform
6 27 Wei Tang
7 26 Wei Tang
* You might need to present your sample request at Analysis Tool Meeting and/or Data Management Meeting
8 27 Wei Tang
9 1 Lu Ren
* The production is ONLY effective with physics conveners' approval. 
10 27 Wei Tang
11 1 Lu Ren
* The fcl files for the production are prepared by user/group who requests the sample and they are also responsible for making sure the production workflow is correct.
12 27 Wei Tang
13 1 Lu Ren
* The user/group are also responsible for validating the sample produced by production team that reaches their expectations. 
14 1 Lu Ren
15 27 Wei Tang
16 27 Wei Tang
h2. Coordination for a large data sample pre-staging
17 27 Wei Tang
18 27 Wei Tang
MicroBooNE requires the pre-staging of dataset larger than "dev" sample (4000 files or 10 TB file size) to be coordinated with production team.  
19 27 Wei Tang
20 27 Wei Tang
* Before making pre-staging request, check pre-staging dataset status table if your requested sample has already been pre-staged. How long the pre-staging last for the dataset depends on various factors. A rule of thumb is if the dataset has been pre-staged within a couple of weeks, then most likely you don't have to pre-stage it. 
21 27 Wei Tang
22 27 Wei Tang
* Check the dataset information with samweb commands 
23 27 Wei Tang
   >$ samweb list-files --summary "defname:Your_Dataset_Name"
24 27 Wei Tang
25 27 Wei Tang
Here is an example:
26 27 Wei Tang
27 27 Wei Tang
samweb list-files --summary "defname:prodgenie_bnb_intrinsic_nue_cosmic_uboone_mcc8.7_reco2" 
28 28 Wei Tang
File count:    12287           ({font color|green|file numbers})
29 27 Wei Tang
Total size:    34235530158207     (file size in byte)
30 27 Wei Tang
Event count:    614350       (available events)
31 27 Wei Tang
32 27 Wei Tang
33 5 Lu Ren
h2. Recommended Data and MC Samples for analysis (as of Aug. 20th, 2018)
34 2 Lu Ren
35 12 Lu Ren
36 12 Lu Ren
Data:
37 12 Lu Ren
* "Good Run Selected Data for 2018":https://microboone-exp.fnal.gov/at_work/AnalysisTools/data/ub_data_2018.html 
38 12 Lu Ren
39 12 Lu Ren
MC:
40 1 Lu Ren
* "MCC8.7 Central Value MC":https://microboone-exp.fnal.gov/at_work/AnalysisTools/mc/mcc8.7/details.html
41 13 Lu Ren
* "MCC8.8 NuMI MC":https://microboone-exp.fnal.gov/at_work/AnalysisTools/mc/mcc8.8/ub_mc_numi.html
42 12 Lu Ren
* "Detector variation samples":https://microboone-exp.fnal.gov/at_work/AnalysisTools/mc/mcc8.10/det_syst.html
43 12 Lu Ren
44 9 Lu Ren
h2. Status of Dataset Prestaging
45 8 Lu Ren
46 7 Lu Ren
|_. Dataset |_. # of files|_. Size|_. Status|_. Requested by |_. Approved|_. Notes |
47 2 Lu Ren
| |  |  | | | | |
48 2 Lu Ren
49 1 Lu Ren
h2. Data and MC samples
50 1 Lu Ren
51 12 Lu Ren
Find full list of samples at "MicroBooNE at Work":https://microboone-exp.fnal.gov/at_work/AnalysisTools/index.html
52 6 Lu Ren
53 23 Anna Mazzacane
h2. List of Production Requests 
54 23 Anna Mazzacane
55 23 Anna Mazzacane
Find actual list of production requests to the team at "List of Production Requests":https://docs.google.com/spreadsheets/d/1yr2KmuzlnLtoEWzTxqd_SWW_mKUluFaGUYXWbv4h5wM/edit#gid=1994969007https://docs.google.com/spreadsheets/d/1yr2KmuzlnLtoEWzTxqd_SWW_mKUluFaGUYXWbv4h5wM/edit#gid=1994969007
56 23 Anna Mazzacane
57 6 Lu Ren
h2. For production team members
58 1 Lu Ren
59 1 Lu Ren
[[How-to's for new team members]]
60 1 Lu Ren
[[Offline Production shift]]
61 1 Lu Ren
"MCC9 Production Plan":https://cdcvs.fnal.gov/redmine/projects/uboone-physics-analysis/wiki/MCC9_Production_Plan
62 1 Lu Ren
[[Others]]
63 1 Lu Ren
64 1 Lu Ren
h2. For users
65 27 Wei Tang
66 27 Wei Tang
Before submitting large number of jobs to the grid, make sure you have tested your workflow following the "Best Practices":https://microboone-docdb.fnal.gov/cgi-bin/private/RetrieveFile?docid=14184&filename=uboone_grid_feb27_2018.pdf&version=2
67 27 Wei Tang
Please "contact the production team":mailto:microboone_offline_production@listserv.fnal.gov, if you 
68 27 Wei Tang
* Want to prestage and process a dataset with more than 4000 files or size > 10 TB
69 27 Wei Tang
* Cancel large number (>100) of jobs on the grid
70 27 Wei Tang
* Have questions about running jobs on the grid