Bug 3507 - cannot list with jglobus 1.2
: cannot list with jglobus 1.2
Status: RESOLVED FIXED
: GridFTP
GridFTP
: 4.0.0
: All Linux
: P3 blocker
: 4.0.1
Assigned To:
:
:
:
:
  Show dependency treegraph
 
Reported: 2005-06-21 12:21 by
Modified: 2005-07-19 16:43 (History)


Attachments


Note

You need to log in before you can comment on or make changes to this bug.


Description From 2005-06-21 12:21:08
With the deprecation of LIST -d, we can no longer list using gridftp 4.0 -- 
this is a real problem for user interface developers such as gateways, portals, 
and other guis that need to do a list of remote directories using tools built 
on the jglobus libraries.
------- Comment #1 From 2005-06-22 15:38:29 -------
Can you give us a list of what options you need?  We dont pass through to the
OS
and then pipe the output back.  We do the stat and build the response ourself,
so we dont want to support every ls option in the book.  We will try and get
the
basic ones in and then let other people contribute code for other options if
they want it supported.

Bill
------- Comment #2 From 2005-06-22 16:58:33 -------
FYI, the Java ftp library is hardcoded with 'LIST -d *' and 'LIST -d <user 
arg>' (where user arg is supposed to be a filter) commands. So I think at least 
the '-d' needs to be handled (e.g. ignored) and basic filtering needs to 
supported (e.g. match all).
------- Comment #3 From 2005-06-28 17:47:37 -------
We will ignore -lad options (our output mimics this already) and * for 4.0.1.   
I would be hard to convince to add any filters that actually require filtering, 
though.
------- Comment #4 From 2005-07-05 15:38:50 -------
I think we need to consider what is important for interversion compatibility -- 
ie:
a) wire protocols (eg, gram, gridftp)
b) low level client apis (eg, jglobus)
c) high level client apis (cog4, trebuchet)
d) service functionality (eg, job manager scripts in gram, this is a 
continued source of pain)
In this case, a) changed -- and we are looking at really bad ways to handle the 
problem in either b) or c).  I think the easiest way to handle it is that we 
preserve the low level client apis, don't have to force changes on the high 
level clients such as cog-4 or trebuchet, and make sure that when the low level 
client issues a list-d, we get back a useful response.  I really don't see 
anyway around it that isn't really ugly.  I do think we need to begin 
considering the impact of changes in these low level services in terms of 
impact to the 4 items I identified (protocol, low level api, high level api, 
service functionality), anytime we induce a change on any of these we have 
impacts to folks building code against the client libraries (not to mention 
making them a wee bit grumpy).
------- Comment #5 From 2005-07-05 16:34:38 -------
How is this handled with the c client libraries?  Somehow programs like uberftp 
can provide listing against gt2 clients, but the code author (Jason Alt) has 
not modified his code.  And does the new uberftp -- interoperate with older 
servers?

Just curious.
------- Comment #6 From 2005-07-05 16:36:39 -------
The c client follows the spec.  Directory list processing is the clients job.  
The server should only expect one (or none) argument to the LIST command, and 
that is the path to list.
------- Comment #7 From 2005-07-05 17:16:12 -------
We are closing this bug as resolved.  Our default output from LIST is
equivalent
to LIST -lad *.  We essentially ignore any options that are passed in, so you
can't do sorting, etc..  So, LIST -d * will work, LIST -d filename will work,
but LIST -d file???.dat will not, LIST -S will not sort, etc..

I want to be clear about a few things:
1) This was NOT a deprecation.  You were taking advantage of non-standard
extensions provided by wuftpd.  RFC 959 does NOT provide for this
functionality.
 Every presentation that has been done about the new server very clearly states
that we would not commit to supporting wuftpd extensions.  We added this one
because an important user community asked us to.

2) We ARE very sensitive to protocol changes.  There have been exactly two in
the 4-5 years GridFTP has been around.  

One was caused by a security bug, so we had no choice.  

This one was caused by the fact that the wuftpd supported non-standard
extensions.  

3) This DOES break protocol compatibility.  RFC 959 allows for spaces in a file
name.  So for instance, FTP should be able to move a file named
-d<space>something.  This will not work now, because we will ignore the -d and
look for a file named something.  You can not solve this by adding double
quotes, they will be considered part of the filename.

This will be available in 4.0.1 or you can get it from CVS using the
globus_4_0_branch tag.
------- Comment #8 From 2005-07-14 12:45:38 -------
After further disucssion, this bug will remain as resolved, but this is a
further update to it.

First, for the general public 3.9.x and the 4.0.0 server will NOT support LIST
-d *.  2.x servers do, and 4.0.1 and greater will because of this fix.

Second, for the general public:  You should *REALLY* use the MLSD functions.  It
is a much cleaner, easier to parse, IETF draft specified format.

This is primarily a TeraGrid issue, and due to a number of issues, TeraGrid will
almost certainly end up having to deploy 4.0.1, meaning that for them, this fix
resolves the issue.

We will update the documentation for the java list functions indicating that
they do not work with 4.0.0.
------- Comment #9 From 2005-07-14 15:30:57 -------
I updated the javadocs for the list() functions with the appropriate info.
------- Comment #10 From 2005-07-19 11:06:11 -------
It appears that LIST -d .* is not functioning.  I paste here the last email
exchange related to this:

If you take a look at bug 3465:

http://bugzilla.globus.org/bugzilla/show_bug.cgi?id=3465

In 4.0.1 we added a SITE VERSION command.  It works like this:

SITE VERSION
200 2.1 (gcc32dbg, 1117216335-1)

gridftp server version (flavor, dirt timestamp-branch tag)

This is the same info that is in the default 220 banners.

You could try that command (or do HELP and search the output).  If it succeeds,
you know you have 4.0.1 or greater, if it fails, you can assume (given your
environment) that any other server will respond as you expect.  This of course
would not protect you from a 4.0.0 server, but again, given your environment,
you *shouldn't* run into one.

The other alternative is to byte the bullet and add support for MLSD. That will
fix the problem permanently and for all servers.  The FEAT command will indicate
whether or not MLST/D is supported, and if so, you can use that, if not, use
your existing code.

HELP OUTPUT (ignore the timestamps):

[10:43:29]: HELP SITE
[10:43:29]: 214-Help for SITE:
[10:43:29]:  SITE SBUF: set send and receive buffers
[10:43:29]:  SITE RETRBUFSIZE: set receive buffers
[10:43:29]:  SITE STORBUFSIZE: set send buffers
[10:43:29]:  SITE RBUFSZ: set receive buffers
[10:43:29]:  SITE RBUFSIZ: set receive buffers
[10:43:29]:  SITE SBUFSZ: set send buffers
[10:43:29]:  SITE SBUFSIZ: set send buffers
[10:43:29]:  SITE HELP: help on server commands
[10:43:29]:  SITE FAULT: force a failure on given command
[10:43:29]:  SITE CHMOD <sp> mode <sp> pathname
[10:43:29]:  SITE RDEL <sp> pathname
[10:43:29]:  SITE DSI <sp> dsi name
[10:43:29]:  SITE VERSION
[10:43:29]: 214 End.

Does that sound reasonable?

Bill

Albert L. Rossi wrote:

> At 09:28 -0500 19/07/2005, Von Welch wrote:
>
>> Appended is the complete log from the last connection to the 4.0.1
>> gridftpd. I've cut what appears to be the relevant lines and pasted
>> them directly below. Obviously something bad is happening but I don't
>> understand what.
>
>
>
>
>> Von
>
>
>
> It is failing on ".*" as an argument.  Obviously, since Bill said it only
supports LIST -d with no arguments.  As I said, we have been doing -d * and -d
.* on the old servers because -d alone does not always return hidden files.
>
> If I  catch and do not propagate the exception, there would then be no way of
determining whether I will have been given hidden files in the case that none
are returned (did the server not give them to me or are there none?).  I think
there will just have to be some other way of determining whether I am talking to
a 4+ server or not.
>
>
>> [21592] Mon Jul 18 15:48:36 2005 :: bi-wireless-149.ncsa.uiuc.edu:4225:
[CLIENT]: LIST -d .*
>> [21592] Mon Jul 18 15:48:36 2005 :: Finished transferring "/home/jalameda/-d .*".
>> [21592] Mon Jul 18 15:48:36 2005 :: bi-wireless-149.ncsa.uiuc.edu:4225:
[SERVER]: 500-Command failed. : System error in stat: No such file or directory
>> 500-A system call failed: No such file or directory
>> 500 End.
>>
>>
------- Comment #11 From 2005-07-19 16:43:28 -------
I've added simple globbing to the server for 4.0.1.  Shell wildcard patterns 
such as .* will now work as expected, and that should be the last of support 
for wuftpd extended listing features that we will add.