Project

General

Profile

Wiki » History » Version 53

« Previous - Version 53/55 (diff) - Next » - Current version
Adam Lister, 10/27/2020 04:35 PM


NOvA Production Wiki


For Users

Production Datasets

Production5 Samples
Production5p1 Samples

Production and You

Current Production Priorities
Submit a Production Request
What's in a Production Request?
Period and Epoch Naming
Production Dataset Naming Conventions

You can contact the production conveners at
------------------------------------------

For Production Shifters

Joining the Production Group

Production Shifts

What to do on production shift
Production Troubleshooting
The Big List Of Production Issues (and Solutions)
Is my sample done?

How to Configure & Run Production Jobs
So You Want to Make Some Monte Carlo (A Qwik Start Guide for Production)
Prestaging datasets whilst on shift
Construct a Draining Dataset
Making concatenated deCAFs
How to respin CAF and deCAF files
Keepup Jobs
Corrupt Files
How to make definitions

Useful Links

Production ECL

Primary tools for monitoring jobs:
Fifemon Grafana
SAM Station
Fifemon FTS
Other more specialized monitoring links
SNOW Quick Links

in addition, running prodjob-summary on any of the gpvms will give a summary of production jobs, eg

$ prodjob-summary 
Total                           Idle    Held    Running Killed  Other   JobName
13066034@jobsub03.fnal.gov      0       0       2       0       0       alister1-reco_nd_detgenie_batch0_rhc_v2-20201010_1923.sh_20201010_192351_2362269_0_1_wrap.sh
37493569@jobsub01.fnal.gov      4999    0       1       0       0       lasquith-ND_rock_fhc_batch2-20201014_1639.sh_20201014_164119_94308_0_1_wrap.shh
===================================================================
Cluster ID                      4999    0       6       0       0


For Conveners

New Convener Onboarding

Convener References

Production Requests
Current Producers
Checking Memory and Time for Jobs
Cleaning Persistent Storage
Dealing with Production Files
End of Epoch Checklist
Making decafs
Making Simulation fcls
Production Campaign Checklist
Production Grid Job User Priority
Publishing New novaproduction and NovaGridUtils releases
Rock Rate Calculation
Submitting Simulation fcls
Using SAM


Other

Running Cosmic Filtering
Deprecated production stuff