Bugzilla – Full Text Bug Listing
|Summary:||Globus-url-copy client does not reuse the connection when run in 3rd party mode.|
|Product:||GridFTP||Reporter:||Gaurang Mehta <firstname.lastname@example.org>|
|Component:||globus-url-copy||Assignee:||Mike Link <email@example.com>|
|Severity:||blocker||CC:||firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org|
|Attachments:||patch to gridftp/client/source/globus_ftp_client_handle.c|
So it seems that from the time the -f <file> option for bulk transfers was introduced the client has not been reusing the connection that the server keeps alive for transferring additional files. (I tested this with versions from 3.2, 3.2.1, 4.0 and 4.0.1 as well as the advisory patch for 4.0.1). How to recreate the scenario. set the limit of instances to 1 or 2 in the xinetd file for gridftp create a input file with more the 2 transfers from and to a gridftp server. invoke the g-u-c from a third site in third party mode globus-url-copy -vb -f <input file> the transfers will start getting an error after the limit in the xinetd has been reached because the server is keeping the initial connections alive for more transfers whereas the client is not reusing them. The client correctly reuses the connection if the same input file is used to transfer in the no third party mode (-notpt) or if one of the url's is a file:// url. This is a major blocker and it would be good if this is fixed and makes it 4.0.2 release.
Does this happen when the 3pt is between two seperate servers?
humm interesting it actually works when the servers are different.
That's a known side-affect of the connection caching, which is keyed on the server contact string. It won't be changed. If you really need to transfer files between the same server for testing purposes or whatever, you can avoid the issue by using the hostname for one and the ip address for the other, or something like that; so long as the actual host:port strings are different. Mike
(In reply to comment #3) > That's a known side-affect of the connection caching, which is keyed on the > server contact string. Is this documented somewhere. It won't be changed. If you really need to transfer > files between the same server for testing purposes or whatever, you can avoid > the issue by using the hostname for one and the ip address for the other, or > something like that; so long as the actual host:port strings are different. > I dont agree that a user should be handling this or worry about why his transfers are failing just cause he chose to do a tpt on the same gridftp server. 1) The error thrown does not tell a user why this happens. I spent nearly half a day figuring out why this was happening. 2) If you can't map 2 connections in your map then you could in the client automatically use the ip for one connection and the hostname for the other rather then expecting the user to do the same 3) IMO the client should continue working albeit inefficently. If you cant do solution 2 then atleast setting notpt automatically on these transfers will also solve the problem. _Gaurang > Mike > (In reply to comment #3) > That's a known side-affect of the connection caching, which is keyed on the > server contact string. It won't be changed. If you really need to transfer > files between the same server for testing purposes or whatever, you can avoid > the issue by using the hostname for one and the ip address for the other, or > something like that; so long as the actual host:port strings are different. > > Mike >
The failures are due to the server configuration, not the client making multiple connections. It would fail the same way if you were only making one connection and happened to have two seperate people trying to connect to the server. I don't think it was documented, but now it is documented here.
let me clarify the situation. Assume, the limit for server connections is 2. I can understand if multiple simultaneous g-u-c instances failed with such a server configuration. but this is not the problem i am describing A single g-u-c should be able to perform multiple transfers from this server to itself in 3rd party mode. But this fails.... whereas it should have succeeded, since this particular client holds both connections to the server. A general application may have no idea that the src and destination urls are on the same host. Note : the first transfer succeeds and the subsequent ones are failing. This is not a consistent behaviour. Either all fail (which IMO should not because it can be trivially handled in the client) or all should succeed (which is what i am rooting for). I dont want to argue for arguing sake just that a user should not have to handle this.
I looked at this further and do see a bug that wasn't expected. (I apoligize if you said this in your earlier comments and I missed it) The client will create a new connection for each file as I expected, but it never closes the old connections, so it uses far more than the 3 connections that I assumed (the extra one that covered the overlap between opening and closing either the source or dest connection). I will fix that, and it should give you the result that a 3pt to the same server only takes 2 connections. This won't make it into 4.0.2 though, as that had already been close to release earlier in the week. Mike
Thanks Mike. Ok so from what you say i understand that while you cannot/or wont handle the reuse of the connections (if the src-dest are same) atleast it wont take more the 2 connections at any given time. Which is fine if not optimized. I am a little worried that it wont make it to 4.0.2 since this would have delivered the fix to a wider audience. I have always found putting advisories on binary installs a pain.
Is there a advisory or patch for this i can build against 4.0.2 It would be good to get this patch and the other one you fixed in the same advisory bundle. -Gaurang
There is no fix for this yet, but it would be a seperate advisory. This is due to a bug in the ftp client library package, not the globus-url-copy package.
Created an attachment (id=1182) [details] patch to gridftp/client/source/globus_ftp_client_handle.c This fixes the connection leak when the source and dest have the same host:port strings.