Project

General

Profile

Feature #7841

Mechanism to specify different environment variables for multiple jobs in one cluster

Added by Christopher Backhouse over 4 years ago. Updated 7 days ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Start date:
02/11/2015
Due date:
% Done:

0%

Estimated time:
Stakeholders:

Nova

Duration:

Description

It's common to submit a lot of jobs whose parameters can't be easily derived by the job script just from their process number. The natural thing to do is figure out the parameters for each job in shell script and then call jobsub_submit providing those parameters as environemnt variables. But I understand that it's better to submit multiple jobs in one Cluster, rather than multiple one-job Clusters. It would be great to have a facility that allows this.

One example of a simple solution is in a simple condor wrapper script I use elsewhere. The user can provide a file, each row corresponding to a job to be submitted, with the environment-variables provided space-seperated. The first line gives the names of the environment variables. eg the file could read:

FOO BAR
baz 1
qux 2

and two jobs would be submitted. One with $FOO=baz, $BAR=1, the other with $FOO=qux, $BAR=2. I used -E for this option to parallel the existing -e for setting an environment variable for all jobs.

-e should continue to work. Calling jobsub_submit -E myfile.txt -e BAZ would submit two jobs, with $BAZ set the same on both, and $FOO and $BAR drawn from myfile.txt

One could additionally add an option that provided command-line arguments for the jobs, rather than environment variables. This would work in the same way but wouldn't require the first line naming the parameters.

Interaction with -N: either ban -E and -N in the same submit command, or have -N x submit x jobs for each single row in the file.

History

#1 Updated by Gavin Davies about 4 years ago

I second this. This would be a great feature to have with the new jobsub_client.

#2 Updated by Parag Mhashilkar about 4 years ago

  • Stakeholders updated (diff)

#3 Updated by Dennis Box 7 days ago

  • Target version set to v1.3.4


Also available in: Atom PDF