Project

General

Profile

Feature #23713

Add an option to skip destination directory check at launch time

Added by Kenneth Herner about 2 months ago. Updated about 2 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
Start date:
12/05/2019
Due date:
% Done:

0%

Estimated time:
Duration:

Description

As we do more and more submission to remote resource (I'm thinking NERSC in this example), we might be writing outputs to local storag at the remote site that isn't visible to us on the gpvm at submit time. For one concrete example, say you're copying back to a directory you made on the scratch filesystem at NERSC, and you're planning to transfer back to FNAL later in some other way. What happens now is that fife_utils checks that the output directory exists and makes it if it doesn't, That's great for most cases, but not this once, since even ifdh ls won't work in that case. So what happens now at submit time is that you spend several minutes waiting for that to fail and retry:

mkdir: cannot create directory ‘/global/cscratch1/sd/dunepro/dunepro_MC_test/nutau’: No such file or directory
Thu Dec 5 11:02:02 2019
program: mkdir /global/cscratch1/sd/dunepro/dunepro_MC_test/nutauexited status 1
delaying 41 ...
Thu Dec 5 11:02:43 2019
retrying...
mkdir: cannot create directory ‘/global/cscratch1/sd/dunepro/dunepro_MC_test/nutau’: No such file or directory

Now eventually it exhausts the retires, gives up, and goes on normally. Since the output directory already exists, everything is fine, except wasting time checking something at submission that will never work. It would be useful to add a command line switch to turn off the destination directory check so we didn't spend time on it. Of course then the onus is on the submitter to make sure the output directories actually exist, but if (s)he has gone to the trouble of specifying the option, that is most likely a safe assumption.

History

#1 Updated by Marc Mengel about 2 months ago

I have change to test in Git; you can set

launch_dest_check: false

in the job_output section, and we won't do the destination check at launch.

In the meantime if your dest variable has '$' in it (i.e $SOME_ENV_VAR or ${metadata_tag}) we won't try to check it at launch 'cause we don't
neceesarily have the variable...



Also available in: Atom PDF