Project

General

Profile

Feature #6342

Allow extra stage(s) in DAG for --dataset_definition

Added by Marc Mengel about 5 years ago. Updated 3 months ago.

Status:
Assigned
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
05/23/2014
Due date:
% Done:

0%

Estimated time:
Stakeholders:
Duration:

Description

Currently the --dataset_definition flag specifies that we run a DAG with
a startproject job, then N workers, and an end project job.

I would like an option to insert another stage in the DAG which
would run a specified script before the endproject job.
This could, for example, use htadd to sum up histogram files
generated by the workers, etc.

Something like --extra_dag_stage=scriptname

which would run that script. I'm currently working on a standalone
script to run DAGs like this, but it would be better to build it
into jobsub.

This can certainly wait for the client/server version.

History

#1 Updated by Parag Mhashilkar about 5 years ago

  • Assignee set to Dennis Box

#2 Updated by Parag Mhashilkar almost 5 years ago

  • Target version set to v1.1

#3 Updated by Dennis Box over 4 years ago

  • Description updated (diff)
  • Status changed from New to Assigned

#4 Updated by Dennis Box over 4 years ago

Hi Marc, Parag,

I see this request is 9 months old, so things may have changed since the original request. It appears to me to be a request to run a user defined POST script on the server. I could see this being a potential security issue.

Marc, I have 2 questions:

1) do you still want this?
2) Would a user defined DAG job that runs on a worker nodes post the users SAM jobs be a better solution?

Thanks
Dennis

#5 Updated by Parag Mhashilkar over 4 years ago

  • Target version changed from v1.1 to v1.2

#6 Updated by Dennis Box over 3 years ago

  • Target version changed from v1.2 to v1.2.2

#7 Updated by Dennis Box about 3 years ago

  • Target version changed from v1.2.2 to v1.2.4

#8 Updated by Dennis Box over 2 years ago

  • Target version changed from v1.2.4 to v1.2.5

#9 Updated by Dennis Box almost 2 years ago

  • Target version changed from v1.2.5 to v1.2.6

#10 Updated by Dennis Box almost 2 years ago

  • Target version changed from v1.2.6 to v1.2.9

#11 Updated by Dennis Box 9 months ago

  • Target version changed from v1.2.9 to v1.3

#12 Updated by Dennis Box 3 months ago

  • Target version changed from v1.3 to v1.3.4


Also available in: Atom PDF