Title:

Allow File Staging To/From globusrun-ws application without an external server

Projects:

VDT, OSG

Technologies:
WS-GRAM, GridFTP

Definition:

The pre-WS globusrun application had a built-in GASS server which allowed users
to submit jobs with file staging and streaming requirements without having
to depend on any external file server. Some user communities would like to have
similar functionality in the globusrun-ws application for WS-GRAM for file
staging.

globusrun-ws will support a new Job description document (JDD) variable,
${GLOBUSRUN_GRIDFTP_URL}.  When globusrun-ws detects this variable in a JDD in
a file staging directive, it will start a gridftp server using the embedded
GridFTP server API.  globusrun-ws will resolve any
variables of the form ${GLOBUSRUN_GRIDFTP_URL} with the URI of
the started server, as well as augment the JDD with the user's DN as
necessary (See examples).

In batch-mode, globusrun-ws will stop the gridftp server running after the
stage_in job state is complete, which could cause a delay in a batch job
submission returning to the command line prompt.  The combination of a batch
job with fileStageOut directives is not supported.  When the job has completed,
the GridFTP server will be stopped.

Example of a user specified JDD using the variable:

    <fileStageIn>
      <transfer>
        <sourceUrl>${GLOBUSRUN_GRIDFTP_SERVER}/tmp/mySourceFile</sourceUrl>
        <destinationUrl>file:///${GLOBUS_USER_HOME}/my_transfered_file</destinationUrl>
      </transfer>
    </fileStageIn>

JDD after globusrun-ws variable processing:

    <fileStageIn>
      <transfer>
        <sourceUrl>gsiftp://job.submitting.host:1234/tmp/mySourceFile</sourceUrl>
        <destinationUrl>file:///${GLOBUS_USER_HOME}/my_transfered_file</destinationUrl>
        <rftOptions>
          <sourceSubjectName>
              /DC=org/DC=doegrids/OU=People/CN=Stuart Martin  564728
          </sourceSubjectName>
        </rftOptions>
      </transfer>
    </fileStageIn>

Deliverables:

- Updated version of globusrun-ws the CVS trunk
- Updated documentation describing enhancement
  - Document globusrun-ws having embedded gridftp server
  - Document new RSL Variables

Tasks:
- Updated globusrun-ws
    - Implement RSL variable handling in C for doing variable substitution in
      RFT-related job elements
    - Add calls to embedded GridFTP server
      - GridFTP server MUST only serve files to client with same credentials
      - GridFTP server should only serve files in the staging lists
- New tests
  - Add tests which submits a job with staging using the new RSL variable and
    globusrun-ws
  - Add batch tests with file stage in with new RSL variables

Dependencies:
- Embedded GridFTP server campaign
  http://bugzilla.mcs.anl.gov/globus/show_bug.cgi?id=4145

Time Estimate:

10 days