Bug 4620 - SEGV on updating credential for GRAM.
: SEGV on updating credential for GRAM.
Status: RESOLVED FIXED
: GRAM
gt2 Gram client
: 4.0.2
: PC Linux
: P3 normal
: 4.0.4
Assigned To:
:
:
:
:
  Show dependency treegraph
 
Reported: 2006-07-26 05:12 by
Modified: 2007-02-14 14:53 (History)


Attachments
gt_refresh_credential.sh (14.48 KB, text/plain)
2006-07-26 05:14, tashiro
Details
GRAM Protocol Patch (3.05 KB, patch)
2006-09-19 11:06, Joe Bester
Details
gt_refresh_credential2.sh (16.36 KB, text/plain)
2006-09-21 00:40, tashiro
Details
GRAM Protocol Patch (3.32 KB, patch)
2006-09-21 19:38, Joe Bester
Details


Note

You need to log in before you can comment on or make changes to this bug.


Description From 2006-07-26 05:12:55
I met the SEGV on GT. conditions are listed below.

  - We created Multi-threaded application, which use GRAM.
  - We use refreshing credential mechanism,
    which execute below functions by individual thread periodically.
      globus_gss_assist_acquire_cred(),
      globus_gram_client_set_credentials()
  - I created the problem reproducing program.
    gt_refresh_credential.sh, which include C program and compile tool.
    I attach it.
  - OS is Linux (RH8), but the problem is occurred on Solaris too.
  - SEGV was occurred on Globus Toolkit 4.0.2.
    SEGV was not occurred on GT 2.4.3.
  - If I do not call globus_gram_client_set_credentials(),
    the SEGV was not occurred.

compile:
$ ./gt_refresh_credential.sh

execute:
$ globus-version
4.0.2
$ globus-hostname > hostlist
$ mv hostlist h ; cat h h h h > hostlist
$ ./gram_cred 1 8 `cat hostlist`
    0 sec: thread 00: start GRAM call to myhost x 8
    0 sec: thread 02: start GRAM call to myhost x 8
    0 sec: thread 01: start GRAM call to myhost x 8
    0 sec: thread 03: start GRAM call to myhost x 8
    0 sec: thread rf: start refresh credential (each 1 sec).
    1 sec: thread 00: 000 start.
    1 sec: thread rf: ref start.
    1 sec: thread 02: 000 start.
    1 sec: thread 01: 000 start.
    1 sec: thread 03: 000 start.
    1 sec: thread rf: ref       end.
Segmentation fault
$
debugger output:

Core was generated by `./gram_cred 1 8 myhost myhost myhost myhost'.
Program terminated with signal 11, Segmentation fault.


#0  0x401ce24d in SSL_new (ctx=0x80629ff) at ssl_lib.c:291
291             if (!s->method->ssl_new(s))
(gdb) where
#0  0x401ce24d in SSL_new (ctx=0x80629ff) at ssl_lib.c:291
#1  0x400fbcd9 in globus_i_gsi_gss_create_and_fill_context (
    minor_status=0x407a187c, context_handle_P=0x407a1884,
    cred_handle=0x80611f0, cred_usage=1, req_flags=4131)
    at globus_i_gsi_gss_utils.c:361
#2  0x400f7370 in gss_init_sec_context (minor_status=0x407a18e4,
    initiator_cred_handle=0x80611f0, context_handle_P=0x80785a8,
    target_name=0x8077438, mech_type=0x0, req_flags=4131, time_req=0,
    input_chan_bindings=0x0, input_token=0x0,
actual_mech_type=0x80785b4,
    output_token=0x407a18dc, ret_flags=0x807859c, time_rec=0x80785a0)
    at init_sec_context.c:128
#3  0x400938d3 in globus_l_xio_gsi_open_cb (op=0x8077fc8, result=0,
    user_arg=0x8078598) at globus_xio_gsi.c:1488
#4  0x400722f4 in globus_l_xio_driver_open_op_kickout
(user_arg=0x8077fc8)
    at globus_xio_driver.c:893
#5  0x4007e7af in globus_xio_driver_finished_open (in_dh=0x8077418,
    in_op=0x8077fc8, in_res=0) at globus_xio_pass.c:231
#6  0x400b331b in globus_l_xio_tcp_system_connect_cb (result=0,
    user_arg=0x80786a0) at globus_xio_tcp_driver.c:1797
#7  0x40084fa0 in globus_l_xio_system_kickout (user_arg=0x8078740)
    at globus_xio_system_select.c:1010
#8  0x401838c9 in globus_l_callback_thread_poll (user_arg=0x401a8a60)
    at globus_callback_threads.c:2482
#9  0x4019bf6c in globus_l_thread_pool_thread_start (user_arg=0x804f988)
    at globus_thread_pool.c:217
#10 0x4019af07 in thread_starter (temparg=0x804bc40)
    at globus_thread_pthreads.c:508
#11 0x40356881 in pthread_detach () from /lib/i686/libpthread.so.0
#12 0x420e40c7 in clone () from /lib/i686/libc.so.6
(gdb)
------- Comment #1 From 2006-07-26 05:14:22 -------
Created an attachment (id=1015) [details]
gt_refresh_credential.sh

The problem reproducing program.
------- Comment #2 From 2006-09-19 11:06:26 -------
Created an attachment (id=1048) [details]
GRAM Protocol Patch

I think this patch corrects this bug. Could you please try it out?
------- Comment #3 From 2006-09-21 00:37:58 -------
Thanks.
I patched my GT 4.0.3 and the program gram_cred was
successfully finished for 4 threads x 8times.
This patch looks fine.

But, unfortunately, My program was still failed.
This program execute globus_gram_client_job_refresh_credentials()
for all available job contacts.

I attach the program using this API. (gt_refresh_credential2.sh)
I executed the following command 10 times.

% globus-hostname > hostlist
% mv hostlist h; cat h h h h > hostlist
% ./gram_cred2 1 8 `cat hostlist`

I got 5 core files.
I always setup MALLOC_CHECK_=2 environment variable on
Linux environment, which affects glibc free().

If I remove this environment variable, the all gram_cred2 finished
successfully and no corefile was output.
(My program was successfully finished too.)

If I use GT 2.4.3 with MALLOC_CHECK_=2,
all gram_cred2 finished successfully too.

All corefiles on GT4.0.3 + patch with MALLOC_CHECK_=2 reports
following stack trace.

Core was generated by `./gram_cred2 1 8
myhost myhost myhost myhost'.
Program terminated with signal 6, Aborted.
#0  0x42028811 in kill () from /lib/i686/libc.so.6
(gdb) where
#0  0x42028811 in kill () from /lib/i686/libc.so.6
#1  0x40358fcd in raise () from /lib/i686/libpthread.so.0
#2  0x420284b4 in raise () from /lib/i686/libc.so.6
#3  0x42029bab in abort () from /lib/i686/libc.so.6
#4  0x42076d89 in posix_memalign () from /lib/i686/libc.so.6
#5  0x42075b85 in free () from /lib/i686/libc.so.6
#6  0x40091701 in globus_l_xio_gsi_handle_destroy (handle=0x81ba400)
    at globus_xio_gsi.c:695
#7  0x4009478a in globus_l_xio_gsi_close (driver_specific_handle=0x81ba400,
    attr=0x0, op=0x8147240) at globus_xio_gsi.c:1801
#8  0x40071b20 in globus_i_xio_driver_start_close (op=0x8147240, can_fail=0)
    at globus_xio_driver.c:702
#9  0x4008272d in globus_xio_driver_read_delivered (op=0x8097e98, in_ndx=0,
    deliver_type=0x407a181c) at globus_xio_pass.c:1506
#10 0x400716a1 in globus_l_xio_driver_op_read_kickout (user_arg=0x8097e98)
    at globus_xio_driver.c:626
#11 0x40081965 in globus_xio_driver_finished_read (in_op=0x8097e98,
    result=662, nbytes=0) at globus_xio_pass.c:1227
#12 0x40095297 in globus_l_xio_gsi_read_cb (op=0x8097e98, result=662,
    nbytes=0, user_arg=0x81ba400) at globus_xio_gsi.c:2063
#13 0x40071688 in globus_l_xio_driver_op_read_kickout (user_arg=0x8097e98)
    at globus_xio_driver.c:620
#14 0x40081965 in globus_xio_driver_finished_read (in_op=0x8097e98,
    result=661, nbytes=0) at globus_xio_pass.c:1227
#15 0x400b45da in globus_l_xio_tcp_finish_read (handle=0x8157d48, result=661,
    nbytes=0) at globus_xio_tcp_driver.c:2196
#16 0x400b4663 in globus_l_xio_tcp_system_read_cb (result=661, nbytes=0,
    user_arg=0x8157d48) at globus_xio_tcp_driver.c:2211
#17 0x40084fea in globus_l_xio_system_kickout (user_arg=0x817d140)
    at globus_xio_system_select.c:1016
#18 0x401838c9 in globus_l_callback_thread_poll (user_arg=0x401a8a60)
    at globus_callback_threads.c:2482
#19 0x4019bf6c in globus_l_thread_pool_thread_start (user_arg=0x804ffc0)
    at globus_thread_pool.c:217
#20 0x4019af07 in thread_starter (temparg=0x804c300)
    at globus_thread_pthreads.c:508
#21 0x40356881 in pthread_detach () from /lib/i686/libpthread.so.0
#22 0x420e40c7 in clone () from /lib/i686/libc.so.6
(gdb)

Thanks.
------- Comment #4 From 2006-09-21 00:40:38 -------
Created an attachment (id=1053) [details]
gt_refresh_credential2.sh

This is the new error reproducing program, 
which use globus_gram_client_job_refresh_credentials().
------- Comment #5 From 2006-09-21 19:38:49 -------
Created an attachment (id=1056) [details]
GRAM Protocol Patch

Attaching a new version of this patch. This catches the case where odd things
can happen when EOF occurs during delegation. There's a similar patch for bug
#4726 which might also help.

joe
------- Comment #6 From 2006-09-22 03:27:54 -------
Thanks for your quick fix.

This patch (with Bug 4726) fixed the problem.
The second program gram_cred2 finished successfully by
4 threads, 64times for each run, times 10 runs.

My original program was no-error too.

This patch is sufficient for me.

Thanks.
------- Comment #7 From 2006-09-22 08:48:41 -------
I've committed (a slightly modified version of) the patch to the 4.0 branch and
CVS trunk. The advisory/update package is available at
http://www.globus.org/toolkit/advisories.html?version=4.0#globus_gram_protocol-6.5

joe
------- Comment #8 From 2006-10-02 18:58:40 -------
*** Bug 4689 has been marked as a duplicate of this bug. ***