Bug List: (This bug is not in your last search results)   Show last search results
Bug 6359 - LIGO: globus-url-copy fails with corrupted double-linked list on CentOS 5.2
: LIGO: globus-url-copy fails with corrupted double-linked list on CentOS 5.2
Status: RESOLVED FIXED
: GridFTP
globus-url-copy
: 4.2.0
: PC Linux
: P3 critical
: ---
Assigned To:
:
:
:
:
  Show dependency tree / graph
 
Reported: 2008-09-05 13:57 by
Modified: 2008-10-22 09:34 (History)

Attachments
Add an attachment (proposed patch, testcase, etc.)


Note

You need to log in before you can comment on or make changes to this bug.

Description From 2008-09-05 13:57:22 
I installed Globus 4.2 on a Cent OS 5.2 x86_64 box and when I run this command

 globus-url-copy -vb -p 4 gsiftp://nemo-dataserver.phys.uwm.edu:15000/dev/zero
gsiftp://dataserver.phy.syr.edu:15000/dev/null

I see this output:

*** glibc detected *** globus-url-copy: corrupted double-linked list:
0x000000000d521600 ***
======= Backtrace: =========
/lib64/libc.so.6[0x346407151c]
/lib64/libc.so.6(cfree+0x8c)[0x3464074ccc]
/opt/LDR/globus/lib/libglobus_gssapi_gsi_gcc64.so.0(gss_release_buffer+0x25)[0x2b6119d5d765]
/opt/LDR/globus/lib/libglobus_ftp_control_gcc64.so.0[0x2b611951524c]
/opt/LDR/globus/lib/libglobus_ftp_control_gcc64.so.0[0x2b6119515526]
/opt/LDR/globus/lib/libglobus_io_gcc64.so.0[0x2b611973b8a6]
/opt/LDR/globus/lib/libglobus_xio_gcc64.so.0(globus_l_xio_read_write_callback_kickout+0x8b)[0x2b611adbb6a7]
/opt/LDR/globus/lib/libglobus_xio_gcc64.so.0(globus_i_xio_read_write_callback+0xe8)[0x2b611adbb85a]
/opt/LDR/globus/lib/libglobus_xio_gcc64.so.0(globus_l_xio_driver_op_read_kickout+0xf4)[0x2b611adbf597]
/opt/LDR/globus/lib/libglobus_xio_gcc64.so.0(globus_xio_driver_finished_read+0x1c3)[0x2b611adc54d0]
/opt/LDR/globus/lib/libglobus_xio_gcc64.so.0[0x2b611addc014]
/opt/LDR/globus/lib/libglobus_xio_gcc64.so.0[0x2b611addc25f]
/opt/LDR/globus/lib/libglobus_xio_gcc64.so.0[0x2b611adc79c1]              
/opt/LDR/globus/lib/libglobus_common_gcc64.so.0(globus_callback_space_poll+0x24e)[0x2b611b60d9d0]
globus-url-copy[0x4063ae]
globus-url-copy(main+0x2da7)[0x409509]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x346401d8b4]
globus-url-copy[0x4041b9]
======= Memory map: ========
00400000-0040e000 r-xp 00000000 fd:00 38568017                          
/opt/LDR/globus/bin/globus-url-copy
0060d000-0060f000 rw-p 0000d000 fd:00 38568017                          
/opt/LDR/globus/bin/globus-url-copy
0060f000-00610000 rw-p 0060f000 00:00 0
0d4bf000-0d5ef000 rw-p 0d4bf000 00:00 0
3463000000-346301a000 r-xp 00000000 fd:00 50855954                      
/lib64/ld-2.5.so
346321a000-346321b000 r--p 0001a000 fd:00 50855954                      
/lib64/ld-2.5.so
346321b000-346321c000 rw-p 0001b000 fd:00 50855954                      
/lib64/ld-2.5.so
3463800000-3463815000 r-xp 00000000 fd:00 50856191                      
/lib64/libselinux.so.1
3463815000-3463a15000 ---p 00015000 fd:00 50856191                      
/lib64/libselinux.so.1
3463a15000-3463a17000 rw-p 00015000 fd:00 50856191                      
/lib64/
libselinux.so.1
3463a17000-3463a18000 rw-p 3463a17000 00:00 0
3463c00000-3463c3b000 r-xp 00000000 fd:00 50856185                      
/lib64/libsepol.so.1
3463c3b000-3463e3b000 ---p 0003b000 fd:00 50856185                      
/lib64/libsepol.so.1
3463e3b000-3463e3c000 rw-p 0003b000 fd:00 50856185                      
/lib64/libsepol.so.1
3463e3c000-3463e46000 rw-p 3463e3c000 00:00 0
3464000000-346414a000 r-xp 00000000 fd:00 50855956                      
/lib64/libc-2.5.so
346414a000-3464349000 ---p 0014a000 fd:00 50855956                      
/lib64/libc-2.5.so
3464349000-346434d000 r--p 00149000 fd:00 50855956                      
/lib64/libc-2.5.so
346434d000-346434e000 rw-p 0014d000 fd:00 50855956                      
/lib64/libc-2.5.so
346434e000-3464353000 rw-p 346434e000 00:00 0
3464400000-3464482000 r-xp 00000000 fd:00 50855962                      
/lib64/libm-2.5.so
3464482000-3464681000 ---p 00082000 fd:00 50855962                      
/lib64/libm-2.5.so
3464681000-3464682000 r--p 00081000 fd:00 50855962                      
/lib64/libm-2.5.so                                                           
3464682000-3464683000 rw-p 00082000 fd:00 50855962                      
/lib64/libm-2.5.so
3464800000-3464802000 r-xp 00000000 fd:00 50855964                      
/lib64/libdl-2.5.so
3464802000-3464a02000 ---p 00002000 fd:00 50855964                      
/lib64/libdl-2.5.so
3464a02000-3464a03000 r--p 00002000 fd:00 50855964                      
/lib64/libdl-2.5.so
3464a03000-3464a04000 rw-p 00003000 fd:00 50855964                      
/lib64/libdl-2.5.so
3465000000-3465014000 r-xp 00000000 fd:00 44637979                      
/usr/lib64/libz.so.1.2.3
3465014000-3465213000 ---p 00014000 fd:00 44637979                      
/usr/lib64/libz.so.1.2.3
3465213000-3465214000 rw-p 00013000 fd:00 44637979                      
/usr/lib64/libz.so.1.2.3
3469c00000-3469c0d000 r-xp 00000000 fd:00 50856153                      
/lib64/libgcc_s-4.1.2-20080102.so.1
3469c0d000-3469e0d000 ---p 0000d000 fd:00 50856153                      
/lib64/libgcc_s-4.1.2-20080102.so.1
3469e0d000-3469e0e000 rw-p 0000d000 fd:00 50856153                      
/lib64/libgcc_s-4.1.2-20080102.so.1
346da00000-346da11000 r-xp 00000000 fd:00 50856174                      
/lib64/libresolv-2.5.so
346dc13000-346dc15000 rw-p 346dc13000 00:00 0
346de00000-346df25000 r-xp 00000000 fd:00 50856221                      
/lib64/
libcrypto.so.0.9.8b
346df25000-346e125000 ---p 00125000 fd:00 50856221                      
/lib64/
libcrypto.so.0.9.8b
346e125000-346e144000 rw-p 00125000 fd:00 50856221                      
/lib64/
libcrypto.so.0.9.8b
346e144000-346e148000 rw-p 346e144000 00:00 0
346e200000-346e202000 r-xp 00000000 fd:00 50856220                      
/lib64/
libcom_err.so.2.1
346e202000-346e401000 ---p 00002000 fd:00 50856220                      
/lib64/
libcom_err.so.2.1
346e401000-346e402000 rw-p 00001000 fd:00 50856220                      
/lib64/
libcom_err.so.2.1
346e600000-346e602000 r-xp 00000000 fd:00 50856173                      
/lib64/
libkeyutils-1.2.so
346e602000-346e801000 ---p 00002000 fd:00 50856173                      
/lib64/
libkeyutils-1.2.so
346e801000-346e802000 rw-p 00001000 fd:00 50856173                      
/lib64/
libkeyutils-1.2.so
346ea00000-346ea24000 r-xp 00000000 fd:00 44643908                      
/usr/li
b64/libk5crypto.so.3.1
346ea24000-346ec23000 ---p 00024000 fd:00 44643908                      
/usr/lib64/libk5crypto.so.3.1                               
346f200000-346f208000 r-xp 00000000 fd:00 44643905                      
/usr/lib64/libkrb5support.so.0.1
346f208000-346f407000 ---p 00008000 fd:00 44643905                      
/usr/lib64/libkrb5support.so.0.1
346f407000-346f408000 rw-p 00007000 fd:00 44643905                      
/usr/lib64/libkrb5support.so.0.1
346f600000-346f62c000 r-xp 00000000 fd:00 44645539                Aborted
------- Comment #1 From 2008-09-05 14:30:57 ------- 
Note that this deployment was built from GT 4.2 source on a Cent OS 5.2 x86_64
box as an RPM. The SPEC file includes this:

%build
./configure --prefix=$RPM_BUILD_ROOT/opt/LDR/globus --with-flavor=gcc64
--with-unixodbc-includes=/usr/include --with-unixodbc-libs=/usr/lib64
--with-buildopts="--verbose" 

make common
make globus-gsi
make gridftp
make globus-rls-server

%install
make install
------- Comment #2 From 2008-09-05 14:52:22 ------- 
Is it possible to get a valgrind report?
------- Comment #3 From 2008-09-05 15:05:52 ------- 
A valgrind report is available at

http://www.lsc-group.phys.uwm.edu/lscdatagrid/downloads/ldr_software/guc-valgrind.out.12631
------- Comment #4 From 2008-09-05 15:42:55 ------- 
A second valgrind report from valgrind run with the --error-limit=no option is
at

http://www.lsc-group.phys.uwm.edu/lscdatagrid/downloads/ldr_software/guc-valgrind.out.13017
------- Comment #5 From 2008-09-05 15:51:36 ------- 
That one actually shows a valgrind failure.. Sorry, can you try it once more
with valgrind opt --undef-value-errors=no

Mike
------- Comment #6 From 2008-09-05 16:11:47 ------- 
Another valgrind output with the options --error-limit=no
--undef-value-errors=no is at

http://www.lsc-group.phys.uwm.edu/lscdatagrid/downloads/ldr_software/guc-valgrind.out.13237
------- Comment #7 From 2008-09-17 15:53:50 ------- 
I used the patch from

http://www-unix.mcs.anl.gov/~mlink/bugs/globus_ftp_control-2.9.tar.gz

and built a new RPM but after installing it I still have problems:

[ldr@dataserver log]$ globus-url-copy -vb -p 4
gsiftp://nemo-dataserver.phys.uwm.edu:15000/dev/zero
gsiftp://dataserver.phy.syr.edu:15000/dev/null
Source: gsiftp://nemo-dataserver.phys.uwm.edu:15000/dev/
Dest:   gsiftp://dataserver.phy.syr.edu:15000/dev/
  zero  ->  null

Segmentation faults        48.53 MB/sec avg        48.53 MB/sec inst

A new valgrind report is at

http://www.lsc-group.phys.uwm.edu/lscdatagrid/downloads/ldr_software/guc-valgrind.out.18091
------- Comment #8 From 2008-09-17 15:55:59 ------- 
I should also add that I reproduced this on another CentOS 5 x86_64 box in
order to rule out any type of memory corruption.
------- Comment #9 From 2008-09-18 14:50:35 ------- 
Thanks for the help in tracking this down.  Looks like openssl on some distros
compresses when it encrypts, which our libs weren't expecting.  patches
committed to HEAD, globus_4_0_branch, and globus_4_2_branch.

Advisories:
http://www-unix.mcs.anl.gov/~mlink/bugs/globus_gridftp_server_control-0.34.tar.gz
http://www-unix.mcs.anl.gov/~mlink/bugs/globus_ftp_control-2.10.tar.gz
------- Comment #10 From 2008-10-22 08:53:22 ------- 
Hi Mike,

I would like to apply this advisory for Globus 4.0.8. I did a "diff" of the
advisories against the 4.0.8 source code, and it looks very different from your
commits (that I found via Bonsai). It looks like the advisories are for GT 4.2.
Is that correct? 

Is the best way for me to get this patch simply to do a diff from CVS instead
of using the advisories?

Thanks,
-alain
------- Comment #11 From 2008-10-22 09:34:20 ------- 
Yeah, these were for 4.2.  Some parts of them were already in 4.0, others might
not build against 4.0.  Note that the problem with openssl compressing anything
shouldn't affect 4.0 either.

This diff should be the entirety of the fixes:
cvs rdiff -kk -u -r globus_4_0_8 -r globus_4_0_branch gridftp/control
gridftp/server-lib
Bug List: (This bug is not in your last search results)   Show last search results