Campaign split type byrun should check if there are any files in run dataset before submitting
Campaign split type byrun should check if there are any files in the split dataset before attempting to submit jobs. If there are no files, poms should automatically advance to the next run. It should keep doing this until it finds a nonempty dataset, or it runs out of runs.
#1 Updated by Herbert Greenlee almost 3 years ago
This update is a clarification of the kinds of new features that we want for POMS.
The issue that we want to solve is that there needs to be a way to run campaigns that avoid merging data accross runs, such that any given batch job should only receive data from a single run. The way that POMS handles this currently, by generating run-specific sam input datasets, is a reasonable approach. However, the current POMS split type byrun, which is based on a once-and-done single pass over a contiguous set of runs, lacks sufficient flexibility. Here are the kinds of additional features we'd like to see.
1. There should be a way to combine split types byrun and draining. I.e., keep submitting a particular run until the input for that run is drained.
2. There should be a way to run campaigns with noncontiguous or sparse input runs. POMS should be taught to figure out which runs have data and which don't, and only submit jobs for which there are input data.