Bug 4344 - Globus-url-copy client does not reuse the connection when run in 3rd party mode.
: Globus-url-copy client does not reuse the connection when run in 3rd party mode.
Status: RESOLVED FIXED
: GridFTP
globus-url-copy
: 4.0.1
: PC Linux
: P3 blocker
: 4.0.4
Assigned To:
:
:
:
:
  Show dependency treegraph
 
Reported: 2006-04-13 16:54 by
Modified: 2007-05-15 15:12 (History)


Attachments
patch to gridftp/client/source/globus_ftp_client_handle.c (2.36 KB, patch)
2007-02-06 20:36, Mike Link
Details


Note

You need to log in before you can comment on or make changes to this bug.


Description From 2006-04-13 16:54:21
So it seems that from the time the -f <file> option for bulk transfers was
introduced the client has not been reusing the connection that the server keeps
alive for transferring additional files. (I tested this with versions from 3.2,
3.2.1, 4.0 and 4.0.1 as well as the advisory patch for 4.0.1).

How to recreate the scenario.

set the limit of instances to 1 or 2 in the xinetd file for gridftp
create a input file with more the 2 transfers from and to a gridftp server.

invoke the g-u-c from a third site in third party mode 

globus-url-copy -vb -f <input file>

the transfers will start getting an error after the limit in the xinetd has
been reached because the server is keeping the initial connections alive for
more transfers whereas the client is not reusing them.

The client correctly reuses the connection if the same input file is used to
transfer in the no third party mode (-notpt) or if one of the url's is a
file:// url.

This is a major blocker and it would be good if this is fixed and makes it
4.0.2 release.
------- Comment #1 From 2006-04-13 17:03:15 -------
Does this happen when the 3pt is between two seperate servers?
------- Comment #2 From 2006-04-13 17:17:47 -------
humm interesting it actually works when the servers are different.
------- Comment #3 From 2006-04-13 17:21:21 -------
That's a known side-affect of the connection caching, which is keyed on the
server contact string.  It won't be changed.  If you really need to transfer
files between the same server for testing purposes or whatever, you can avoid
the issue by using the hostname for one and the ip address for the other, or
something like that; so long as the actual host:port strings are different.

Mike
------- Comment #4 From 2006-04-13 17:39:22 -------
(In reply to comment #3)
> That's a known side-affect of the connection caching, which is keyed on the
> server contact string.  

Is this documented somewhere.

It won't be changed.  If you really need to transfer
> files between the same server for testing purposes or whatever, you can avoid
> the issue by using the hostname for one and the ip address for the other, or
> something like that; so long as the actual host:port strings are different.
> 

I dont agree that a user should be handling this or worry about why his
transfers are failing just cause he chose to do a tpt on the same gridftp
server.

1) The error thrown does not tell a user why this happens. I spent nearly half
a day figuring out why this was happening. 

2) If you can't map 2 connections in your map then you could in the client
automatically use the ip for one connection and the hostname for the other
rather then expecting the user to do the same

3) IMO the client should continue working albeit inefficently. If you cant do
solution 2 then atleast setting notpt automatically on these transfers will
also solve the problem.

_Gaurang


> Mike
> 

(In reply to comment #3)
> That's a known side-affect of the connection caching, which is keyed on the
> server contact string.  It won't be changed.  If you really need to transfer
> files between the same server for testing purposes or whatever, you can avoid
> the issue by using the hostname for one and the ip address for the other, or
> something like that; so long as the actual host:port strings are different.
> 
> Mike
> 
------- Comment #5 From 2006-04-13 17:46:04 -------
The failures are due to the server configuration, not the client making
multiple connections.  It would fail the same way if you were only making one
connection and happened to have two seperate people trying to connect to the
server.

I don't think it was documented, but now it is documented here.
------- Comment #6 From 2006-04-13 18:05:11 -------
let me clarify the situation.

Assume, the limit for server connections is 2.

I can understand if multiple simultaneous g-u-c instances failed with such a
server configuration.

but this is not the problem i am describing

A single g-u-c should be able to perform multiple transfers from this server to
itself in 3rd party mode. 

But this fails.... whereas it should have succeeded, since this particular
client holds both connections to the server.

A general application may have no idea that the src and destination urls are on
the same host. 

Note : the first transfer succeeds and the subsequent ones are failing. This is
not a consistent behaviour. Either all fail (which IMO should not because it
can be trivially handled in the client)  or all should succeed (which is what i
am rooting for).

I dont want to argue for arguing sake just that a user should not have to
handle this.
------- Comment #7 From 2006-04-14 13:44:33 -------
I looked at this further and do see a bug that wasn't expected.  (I apoligize
if you said this in your earlier comments and I missed it) The client will
create a new connection for each file as I expected, but it never closes the
old connections, so it uses far more than the 3 connections that I assumed (the
extra one that covered the overlap between opening and closing either the
source or dest connection).  I will fix that, and it should give you the result
that a 3pt to the same server only takes 2 connections.

This won't make it into 4.0.2 though, as that had already been close to release
earlier in the week.

Mike  
------- Comment #8 From 2006-04-14 15:31:25 -------
Thanks Mike.

Ok so from what you say i understand that while you cannot/or wont handle the
reuse of the connections (if the src-dest are same) atleast it wont take more
the 2 connections at any given time. Which is fine if not optimized.

I am a little worried that it wont make it to 4.0.2 since this would have
delivered the fix to a wider audience. I have always found putting advisories
on binary installs a pain.

------- Comment #9 From 2006-07-07 17:47:18 -------
Is there a advisory or patch for this i can build against 4.0.2
It would be good to get this patch and the other one you fixed in the same
advisory bundle.

-Gaurang
------- Comment #10 From 2006-07-08 01:44:09 -------
There is no fix for this yet, but it would be a seperate advisory.  This is due
to a bug in the ftp client library package, not the globus-url-copy package.
------- Comment #11 From 2007-02-06 20:36:47 -------
Created an attachment (id=1182) [details]
patch to gridftp/client/source/globus_ftp_client_handle.c

This fixes the connection leak when the source and dest have the same host:port
strings.
------- Comment #12 From 2007-05-15 15:12:24 -------
This fix was included in the 4.0.4 release.