Project

General

Profile

globusonline plugin

Documentation

All the relevant documentation has been moved to the Manual which is now available through the git repository.
https://cdcvs.fnal.gov/redmine/projects/go-condor-plugin/repository/revisions/master/changes/doc/Condor-GO-Transfer-Plugin-Manual.odt

Repository

If you have FNAL Kerberos ticket, you can clone the git repo by
git clone ssh:///cvs/projects/go-condor-plugin

Releases

Releases are tagged with the version numbers

v0.1: First Release
v0.2: Oct 05, 2011

Tasks that need to be migrated to glideinWMS

  • Plugin and plugin setup scripts are available independent of glideinWMS releases. In future setup should be part of the glideinWMS release.
    • From email conversation with LBNE, Panda is planning on supporting GO (?). Needs to be confirmed. * In general plugin architecture will be used as a means to interface WMS with the public storage if proposal goes through. Tanya & Gabriele working on this.
  • Follow up with the IF experiments periodically
    • Minerva has shown interest in adopting Globus Online for the transfers * Did load testing with 3000+ jobs and it went quite well * Other IF experiments might follow their steps
  • Follow up with the Globus Online team periodically

    Action items from meeting with Globus team on Aug 25, 2011 -

    • GC accepting incoming connections * GO accepting fall-back endpoint * GO accepting time-out on lifetime of a GC (if job / GC is killed, nobody will remove the end points) * GC configuring the number of accepted GO transfers
  • Follow up with the Condor team periodically and use the new transfer plugin architecture when it is released.
    • Currently, condor plans to release it with Condor 7.9 series * Adapt to new plugin architecture * Make the plugin available as a contrib module in the Condor * Condor transfer plug-in interface accepting fall-back endpoint * Details in assumptions section: X509_USER_PROXY, LOGGING * Need better means to pass more information to the plugins
  • Follow up with the Condor folks on the asynchronous sandbox management and integrate it with the glideinWMS
    • Plugins in general are good way to support custom transfer protocols but they are much more useful when used with asynchronous sandbox management

Plugin: Current Assumptions & Known Issues

  • X509_USER_PROXY
    Currently condor is not consistent with the X509_USER_PROXY. It is correct when called for input file transfers but is different when plugin is called for output file transfers. To get around the problem, just use the first file in the execute dir that starts with x509up_u.
  • GLOBUSCONNECT
    We need to do a better job of figuring out which GLOBUSCONNECT to use. For some reason $GLOBUSCONNECT is not available in plugin's environment.
  • LOGGING
    Better logging and means to unify logs from all GO transfers to a single file. Also, all the files go to the GO endpoints and nothing gets transferred back. So if there is a problem accessing GO itself, logs are lost. You can still get stdout and stderr back to the submit machine if you stream them. But plugin is not a part of the job so that doesn't help.
  • Reusing/Reducing number of Globus Online endpoints
    Currently, each GO file creates a new GO endpoint. We can use condor hooks to start globusconnect once and use it for all the transfers.
  • Supporting multiple transfers
    It is inefficient to transfer one file at a time. It would be nice to club several files together to be transferred in just one command
  • Fail earlier than 1 day default for Globus Online
    Start the transfer in background and periodically poll it. Fail accordingly rather than the default GO deadline of 1d