Tips for doing MC production¶
- The basic premise of a MC production is the same as running an individual sample ( there's just a lot of them! )
- Always use the head of develop i.e. a release version, not your local products
- We want the versions of the code to be free of any local changes.
- Sometimes it is necessary to perform hot fixes, in this case use a tarball having changed only the necessary parts of the code.
- As an MC production involves many samples, it is easier to submit via the command line than using the gui.
- As an MC production involves producing many samples where each will take differing amounts of time a method of tracking progress is needed.
- An easy way of doing this is to use an excel spreadsheet with a line for each sample, and columns for each stage.
- At each stage you need to submit, check and declare the files so it is useful to split each stage column into three.
- It can also be useful to use separate terminal tabs for each sample as then the previous command can be easily checked.
Things to look out for¶
- A stage is usually considered successful if at least 95% of the jobs in that stage completed without an error.
- This is because the samples produced are quite large, and so some losses can be accepted.
- If many jobs are unsuccessful there could be numerous reasons
- A common fault at the reco stage is that the filename is more than 200 characters. This can be fixed with --shorten
- A new configuration parameter may not have correctly set up. This will often be evident from looking at the log files, or attempting to run the script locally.
- A new feature of the code has not been tested rigorously enough, this should be rare as efforts are made to ensure that no major changes are made just before making production samples.
- If many jobs are unsuccessful then they can be resubmitted with the --makeup option.
Process of submitting samples¶
- The first few steps are identical to those that you would do when producing a personal sample
- Setup your desired version of LArSoft - make sure to use prof!!!
setup dunetpc < latest version (v06_16_00) > -q < qualifiers (e10:prof) >
- Ensure that you have a valid ticket
kx509;voms-proxy-init -noregen -rfc -voms dune:/dune/Role=Production
- For each sample then proceed to execute the stages as you would do normally, with one extra step...
project.py --xml ____ --stage ____ --submit
project.py --xml ____ --stage ____ --check
project.py --xml ____ --stage ____ --delcare
- For the mergeana / analysis stage there are some extra steps
- Merge analysis histograms
project.py --xml ____ --stage ____ --mergehist
- Create a samweb dataset definition
project.py --xml ____ --stage ____ --define
- You can check that the dataset is correct
project.py --xml ____ --stage ____ --check_declarations
- Upload the files to enstore
project.py --xml ____ --stage ____ --upload