Project

General

Profile

Bug #24678

User job invocation via singularity eating up -e or -n

Added by Marco Mambelli about 2 months ago. Updated 13 days ago.

Status:
Closed
Priority:
Immediate
Category:
-
Target version:
Start date:
07/30/2020
Due date:
% Done:

0%

Estimated time:
First Occurred:
Occurs In:
Stakeholders:
Duration:

Description

When a user job has an argument/option that is a valid option for echo this is not preserved when using singularity via the GWMS wrapper.
This happens because of a bug in the function fixing the paths:

       arg="`echo "$arg" | sed -E "s,$outside_pwd/(.*),$inside_pwd/\1,;s,.*/execute/dir_[0-9a-zA-Z]*(.*),$inside_pwd\1,"`" 

Echo may think it is an option and 'eats' it

History

#1 Updated by Marco Mambelli about 2 months ago

This is sufficient to fix singularity_update_path not to eat options:

-        arg="`echo "$arg" | sed -E "s,$outside_pwd/(.*),$inside_pwd/\1,;s,.*/execute/dir_[0-9a-zA-Z]*(.*),$inside_pwd\1,"`" 
-        GWMS_RETURN+=("$arg")
+        arg="$(echo -n "" "$arg" | sed -E "s,$outside_pwd/(.*),$inside_pwd/\1,;s,.*/execute/dir_[0-9a-zA-Z]*(.*),$inside_pwd\1,")" 
+        GWMS_RETURN+=("${arg# }")

But would be preferrable to use parameter expansion, not to risk corner cases.

#2 Updated by Marco Mambelli about 2 months ago

Mats Rynge explaining the 'execute/dir_' replacement:

The problem is that sometimes $PWD does not match the actual configured startup dir. I don't know exactly where this goes wrong, but it has to do with symlinks. For example, we have seen things like:

PWD=/filesystem1/glidein/execute/dir_1

while the startup dir was

/scratch/glidein/execute/dir_1

Turns out that /scratch is a symlink to /filesystem1. That extra sed just protects from that corner case, by replacing anything that looks like a HTCondor execute dir with /srv.

#3 Updated by Marco Mambelli about 2 months ago

This solution:
1. uses parameter expansion
2. replaces given path and real path, but not any execute/dir_ path
3. paths must start form the beginning: this protects against subpaths with the same string (e.g. /cvmfs/some/path/here) but is not replacing if the path is not at the beginning of the option (e.g. --file=/path)

   local outside_pwd="${GWMS_SINGULARITY_OUTSIDE_PWD:-$PWD}" 
   local realpath_outside_pwd="$(robust_realpath "${outside_pwd}")" 
   local inside_pwd=$1
...
    arg="${arg/#$outside_pwd/$inside_pwd}" 
    [[ "${realpath_outside_pwd}" != "${outside_pwd}" ]] && arg="${arg/#$realpath_outside_pwd/$inside_pwd}" 
        [[ "${arg}" == *"${outside_pwd}"* || "${arg}" == *"${realpath_outside_pwd}"* ]] && warn "Outside path still in argument path ($arg), the conversion to run in Singularity may be incorrect" 
    [[ "${arg}" = */execute/dir_* ]] && warn "String '/execute/dir_' in argument path ($arg), the conversion to run in Singularity may be incorrect" 
...

#4 Updated by Marco Mambelli about 2 months ago

  • Assignee changed from Marco Mambelli to Dennis Box
  • Status changed from New to Feedback

#5 Updated by Marco Mambelli about 1 month ago

  • Assignee changed from Dennis Box to Marco Mambelli
  • Status changed from Feedback to Resolved

#6 Updated by Marco Mambelli 13 days ago

  • Status changed from Resolved to Closed


Also available in: Atom PDF