Feature #23713

Add an option to skip destination directory check at launch time

Added by Kenneth Herner 7 months ago. Updated 18 days ago.

Target version:
Start date:
Due date:
% Done:


Estimated time:


As we do more and more submission to remote resource (I'm thinking NERSC in this example), we might be writing outputs to local storag at the remote site that isn't visible to us on the gpvm at submit time. For one concrete example, say you're copying back to a directory you made on the scratch filesystem at NERSC, and you're planning to transfer back to FNAL later in some other way. What happens now is that fife_utils checks that the output directory exists and makes it if it doesn't, That's great for most cases, but not this once, since even ifdh ls won't work in that case. So what happens now at submit time is that you spend several minutes waiting for that to fail and retry:

mkdir: cannot create directory ‘/global/cscratch1/sd/dunepro/dunepro_MC_test/nutau’: No such file or directory
Thu Dec 5 11:02:02 2019
program: mkdir /global/cscratch1/sd/dunepro/dunepro_MC_test/nutauexited status 1
delaying 41 ...
Thu Dec 5 11:02:43 2019
mkdir: cannot create directory ‘/global/cscratch1/sd/dunepro/dunepro_MC_test/nutau’: No such file or directory

Now eventually it exhausts the retires, gives up, and goes on normally. Since the output directory already exists, everything is fine, except wasting time checking something at submission that will never work. It would be useful to add a command line switch to turn off the destination directory check so we didn't spend time on it. Of course then the onus is on the submitter to make sure the output directories actually exist, but if (s)he has gone to the trouble of specifying the option, that is most likely a safe assumption.


#1 Updated by Marc Mengel 7 months ago

I have change to test in Git; you can set

launch_dest_check: false

in the job_output section, and we won't do the destination check at launch.

In the meantime if your dest variable has '$' in it (i.e $SOME_ENV_VAR or ${metadata_tag}) we won't try to check it at launch 'cause we don't
neceesarily have the variable...

#2 Updated by Marc Mengel 18 days ago

  • Status changed from New to Closed

There is already a flag for this; in your config, you can set launch_dest_check to False
on each job_output[_n] stanza

  launch_dest_check: False

#3 Updated by Marc Mengel 18 days ago

  • Target version set to v3_3

Also available in: Atom PDF