Mechanism to specify different environment variables for multiple jobs in one cluster
It's common to submit a lot of jobs whose parameters can't be easily derived by the job script just from their process number. The natural thing to do is figure out the parameters for each job in shell script and then call
jobsub_submit providing those parameters as environemnt variables. But I understand that it's better to submit multiple jobs in one Cluster, rather than multiple one-job Clusters. It would be great to have a facility that allows this.
One example of a simple solution is in a simple condor wrapper script I use elsewhere. The user can provide a file, each row corresponding to a job to be submitted, with the environment-variables provided space-seperated. The first line gives the names of the environment variables. eg the file could read:
FOO BAR baz 1 qux 2
and two jobs would be submitted. One with
$FOO=baz, $BAR=1, the other with
$FOO=qux, $BAR=2. I used
-Efor this option to parallel the existing
-efor setting an environment variable for all jobs.
-e should continue to work. Calling
jobsub_submit -E myfile.txt -e BAZ would submit two jobs, with
$BAZ set the same on both, and
$BAR drawn from
One could additionally add an option that provided command-line arguments for the jobs, rather than environment variables. This would work in the same way but wouldn't require the first line naming the parameters.
-N: either ban
-N in the same submit command, or have
-N x submit
x jobs for each single row in the file.