Project

General

Profile

Bug #22020

On POMS integration instance the automatic submission for the second stage is queued/hold

Added by Vito Di Benedetto 11 months ago. Updated 11 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Start date:
02/28/2019
Due date:
% Done:

0%

Estimated time:
First Occurred:
Scope:
Internal
Experiment:
-
Stakeholders:
Duration:

Description

On POMS integration instance I have the Campaign test_vito_tutorial_03; ID: 3277; VO: uboone; poms_role: analysis
https://poms-int.fnal.gov/poms/show_campaign_stages?campaign_name=test_vito_tutorial_03

I manually submit jobs for the first stage (Campaign stage: sim; ID: 2799) , then job submission from the second stage (Campaign stage: reco; ID: 2800), that depends on the first stage are, is queued and I got en email saying:

Due to an invalid proxy, we had to queue a job launch
Please upload a new proxy, and release queued jobs for this campaign

But the proxy uploaded to POMS is still valid, it has been successfully used for another submission.

History

#1 Updated by Marc Mengel 11 months ago

So I'm pretty sure this was the ownership problem, where submissions were showing up as belonging to to me (mengel). I think this is fixed now,but I can't test it, because I can't see the difference between it getting the ownership right (I am mengel) or wrong...

#2 Updated by Vito Di Benedetto 11 months ago

OK, I just submitted 1 job to test the fix.
Though the job is going to fail when the final output is required to get its metadata declared to SAM.
But it looks like POMS tries to submit jobs for the next campaign stage anyway. We will see if the submission will happen or it will be aborted as there are no input files from the previous stage.

#3 Updated by Marc Mengel 11 months ago

which is to say, it was complaining about my proxy, not yours because it thought the submission belonged to me...

#4 Updated by Vito Di Benedetto 11 months ago

So the test jobs went through. As expected it failed.
POMS tried to submit jobs from the second stage, this time the submission attempt succeeded, i.e. the submission was not hold as now the proxy was found.
Then the submission was aborted by POMS, as the input dataset was empty.
It looks like the issue reported here fixed.

#5 Updated by Yuyi Guo 11 months ago

So the second stage was submitted under your name/vito, not Marc?

#6 Updated by Vito Di Benedetto 11 months ago

Yuyi Guo wrote:

So the second stage was submitted under your name/vito, not Marc?

yes, that's correct.
Though the submission was aborted by POMS as there were no files in the input dataset, this was expected.

The submission attempt is available here:
https://poms-int.fnal.gov/poms/list_launch_file?campaign_stage_id=2800&fname=20190301_212804_vito_441598

#7 Updated by Stephen White 11 months ago

  • Assignee set to Marc Mengel

Marc, is this fixed now?

#8 Updated by Marc Mengel 11 months ago

I need someone who is Not Me to run a dependency chain under Analysis to be sure.

#9 Updated by Yuyi Guo 11 months ago

I just submitted a 6-stage campaign. I will let you know what I get when it is done.

#10 Updated by Yuyi Guo 11 months ago

  • Status changed from New to Closed

I confirm this is fixed.



Also available in: Atom PDF