5706 2007-12-06 12:09 globus-url-copy returning "error: No files matched the source url" when asked to copy a directory 2008-01-17 00:28:06 1 1 1 Unclassified GridFTP globus-url-copy unspecified PC Linux RESOLVED FIXED P3 major 4.0.6 1 marco@hep.uchicago.edu mlink@mcs.anl.gov bester@mcs.anl.gov marco@hep.uchicago.edu roy@cs.wisc.edu marco@hep.uchicago.edu 2007-12-06 12:09:42 With newer globus-url-copy the directory parsing is not working and even the copy of one subdirectory is failing My goal was to do a recursive copy using 3rd party transfer. I've been comparing the results of 2 different versions of globus-url-copy that behave differently. with globus-url-copy: 3.18 (1120222988-63) > globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/ gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5f/ is equivalent to globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/* gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5g/ No directory is created, all files are copied but if there are directories these cause problems. This should probably be the normal behavior for interactions with servers that support only NLST (and not the extended ftp listing commands) if I understood correctly the help text. With globus-url-copy: 3.22 (1172269086-63) (any version >= 3.20) I tried the same and I get this error: /share/osg-client > globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/ gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5f/ error: No files matched the source url. The same error is given also by gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/* as source URL Checking the -dbg execution: NLST /pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008 is sent and received OK, but the parsing is returning the error above I have a twiki page with my tests: http://twiki.mwt2.org/bin/view/Main/TroubleShooting071205guc#GridFTP marco@hep.uchicago.edu 2007-12-06 13:49:47 The problem is independent from the '-r' switch as shown below. Furthermore I'm adding globus-url-copy -versions and vdt-version [uct3-edge1] /share/osg-client > globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/ gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5f/ error: No files matched the source url. [uct3-edge1] /share/osg-client > globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/* gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5f/ error: No files matched the source url. [uct3-edge1] /share/osg-client > globus-url-copy gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/* gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5f/ error: No files matched the source url. [uct3-edge1] /share/osg-client > globus-url-copy -versions globus-url-copy: 3.22 (1172269086-63) globus_ftp_client_restart_plugin: 3.6 (1171321665-63) globus_ftp_client_debug_plugin: 3.6 (1171321665-63) globus_ftp_client_perf_plugin: 3.6 (1171321665-63) globus_ftp_client_throughput_plugin: 3.6 (1171321665-63) globus_ftp_control: 2.5 (1181535824-63) globus_ftp_client: 3.6 (1171321665-63) globus_xio_gsi: 0.36 (1163020628-63) globus_xio_tcp: 0.36 (1163020628-63) globus_xio_system_select: 0.36 (1163020628-63) globus_xio_file: 0.36 (1163020628-63) globus_xio: 0.36 (1163020628-63) globus_io: 6.0 (1113865852-1) globus_gsi_callback_module: 0.23 (1121890023-63) globus_credential: 1.9 (1159917212-63) globus_gsi_proxy: 2.4 (1113610195-1) globus_gsi_openssl_error: 0.13 (1170803088-63) globus_openssl: 0.6 (1113610167-1) globus_gsi_gssapi: 4.12 (1179416313-63) globus_sysconfig: 1.20 (1159917212-63) globus_callout_module: 0.6 (1113865156-1) globus_gss_assist: 3.23 (1170694562-63) globus_i_gass_transfer_http: 2.13 (1174421873-63) globus_extension_module: 7.24 (1181664948-63) globus_callback_nonthreaded: 7.24 (1181664948-63) globus_object: 7.24 (1181664948-63) globus_error: 7.24 (1181664948-63) globus_common: 7.24 (1181664948-63) globus_gass_transfer: 2.13 (1174421873-63) globus_gass_copy: 3.22 (1172269086-63) globus_thread_common: 7.24 (1181664948-63) globus_thread_none: 7.24 (1181664948-63) [uct3-edge1] /share/osg-client > vdt-version You have installed a subset of VDT version 1.8.1e: CA Certificates v33 (includes IGTF 1.18 CAs) Condor/Condor-G 6.8.6 cURL 7.16.2 Fetch CRL 2.6.2 Globus Toolkit, pre web-services, client 4.0.5 Globus Toolkit, web-services, client 4.0.5 GPT 3.2 GSI-Enabled OpenSSH 4.0 Java 5 SDK 1.5.0_13 KX509 20031111 lcg-info 1.11.0-1 lcg-infosites 2.6-2 Logrotate 3.7 MyProxy 3.9 Pegasus 2.0.1 PPDG Cert Scripts 2.5 pyGlobus gt4.0.1-1.13 PyGlobus URL Copy 1.1.2.11 RLS, client 3.0.041021 SRM V1 Client 1.25 SRM V2 Client 2.2.0.4 UberFTP 1.24 Wget 1.10.2 mlink@mcs.anl.gov 2007-12-06 13:53:41 Committed a fix to trunk and globus_4_0_branch. Try this update package: http://www-unix.mcs.anl.gov/~mlink/bugs/globus_gass_copy-3.23.tar.gz (use the update instructions at http://www.globus.org/toolkit/advisories.html) mlink@mcs.anl.gov 2007-12-06 13:55:00 You are correct that the 3.18 behavior is normal, this fix restores that behavior.