Bug 5706

Summary: globus-url-copy returning "error: No files matched the source url" when asked to copy a directory
Product: GridFTP Reporter: Marco Mambelli <marco@hep.uchicago.edu>
Component: globus-url-copyAssignee: Mike Link <mlink@mcs.anl.gov>
Status: RESOLVED FIXED    
Severity: major CC: bester@mcs.anl.gov, marco@hep.uchicago.edu, roy@cs.wisc.edu
Priority: P3    
Version: unspecified   
Target Milestone: 4.0.6   
Hardware: PC   
OS: Linux   

Description From 2007-12-06 12:09:42 
With newer globus-url-copy the directory parsing is not working and even the
copy of one subdirectory is failing

My goal was to do a recursive copy using 3rd party transfer.
I've been comparing the results of 2 different versions of globus-url-copy that
behave differently. 

with globus-url-copy: 3.18 (1120222988-63) 
> globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/  gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5f/

is equivalent to globus-url-copy -r
gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/*
gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5g/

No directory is created, all files are copied but if there are directories
these cause problems.
This should probably be the normal behavior for interactions with servers that
support only NLST (and not the extended ftp listing commands) if I understood
correctly the help text.

With globus-url-copy: 3.22 (1172269086-63) (any version >= 3.20) I tried the
same and I get this error:
/share/osg-client > globus-url-copy -r
gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/
gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5f/

error: No files matched the source url.


The same error is given also by
gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/*
as source URL

Checking the -dbg execution:
NLST
/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008
is sent and received OK, but the parsing is returning the error above


I have a twiki page with my tests:
http://twiki.mwt2.org/bin/view/Main/TroubleShooting071205guc#GridFTP
------- Comment #1 From 2007-12-06 13:49:47 ------- 
The problem is independent from the '-r' switch as shown below.
Furthermore I'm adding globus-url-copy -versions and vdt-version

[uct3-edge1] /share/osg-client > globus-url-copy -r
gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/
gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5f/

error: No files matched the source url.

[uct3-edge1] /share/osg-client > globus-url-copy -r
gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/*
gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5f/

error: No files matched the source url.

[uct3-edge1] /share/osg-client > globus-url-copy 
gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/*
gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5f/

error: No files matched the source url.

[uct3-edge1] /share/osg-client > globus-url-copy -versions
globus-url-copy: 3.22 (1172269086-63)
globus_ftp_client_restart_plugin: 3.6 (1171321665-63)
globus_ftp_client_debug_plugin: 3.6 (1171321665-63)
globus_ftp_client_perf_plugin: 3.6 (1171321665-63)
globus_ftp_client_throughput_plugin: 3.6 (1171321665-63)
globus_ftp_control: 2.5 (1181535824-63)
globus_ftp_client: 3.6 (1171321665-63)
globus_xio_gsi: 0.36 (1163020628-63)
globus_xio_tcp: 0.36 (1163020628-63)
globus_xio_system_select: 0.36 (1163020628-63)
globus_xio_file: 0.36 (1163020628-63)
globus_xio: 0.36 (1163020628-63)
globus_io: 6.0 (1113865852-1)
globus_gsi_callback_module: 0.23 (1121890023-63)
globus_credential: 1.9 (1159917212-63)
globus_gsi_proxy: 2.4 (1113610195-1)
globus_gsi_openssl_error: 0.13 (1170803088-63)
globus_openssl: 0.6 (1113610167-1)
globus_gsi_gssapi: 4.12 (1179416313-63)
globus_sysconfig: 1.20 (1159917212-63)
globus_callout_module: 0.6 (1113865156-1)
globus_gss_assist: 3.23 (1170694562-63)
globus_i_gass_transfer_http: 2.13 (1174421873-63)
globus_extension_module: 7.24 (1181664948-63)
globus_callback_nonthreaded: 7.24 (1181664948-63)
globus_object: 7.24 (1181664948-63)
globus_error: 7.24 (1181664948-63)
globus_common: 7.24 (1181664948-63)
globus_gass_transfer: 2.13 (1174421873-63)
globus_gass_copy: 3.22 (1172269086-63)
globus_thread_common: 7.24 (1181664948-63)
globus_thread_none: 7.24 (1181664948-63)

[uct3-edge1] /share/osg-client > vdt-version 
You have installed a subset of VDT version 1.8.1e:
    CA Certificates v33 (includes IGTF 1.18 CAs)
    Condor/Condor-G 6.8.6
    cURL 7.16.2
    Fetch CRL 2.6.2
    Globus Toolkit, pre web-services, client 4.0.5
    Globus Toolkit, web-services, client 4.0.5
    GPT 3.2
    GSI-Enabled OpenSSH 4.0
    Java 5 SDK 1.5.0_13
    KX509 20031111
    lcg-info 1.11.0-1
    lcg-infosites 2.6-2
    Logrotate 3.7
    MyProxy 3.9
    Pegasus 2.0.1
    PPDG Cert Scripts 2.5
    pyGlobus gt4.0.1-1.13
    PyGlobus URL Copy 1.1.2.11
    RLS, client 3.0.041021
    SRM V1 Client 1.25
    SRM V2 Client 2.2.0.4
    UberFTP 1.24
    Wget 1.10.2
------- Comment #2 From 2007-12-06 13:53:41 ------- 
Committed a fix to trunk and globus_4_0_branch.  

Try this update package:
http://www-unix.mcs.anl.gov/~mlink/bugs/globus_gass_copy-3.23.tar.gz
(use the update instructions at http://www.globus.org/toolkit/advisories.html)
------- Comment #3 From 2007-12-06 13:55:00 ------- 
You are correct that the 3.18 behavior is normal, this fix restores that
behavior.