First Last Prev Next    No search results available
Bug 4732 - globusrun-ws segmentation fault (FC5 x86_64)
: globusrun-ws segmentation fault (FC5 x86_64)
Status: RESOLVED FIXED
: GRAM
wsrf gram clients
: 4.0.3
: Other Linux
: P1 normal
: 4.0.4
Assigned To:
:
:
:
:
  Show dependency tree / graph
 
Reported: 2006-09-26 11:37 by
Modified: 2006-11-06 10:36 (History)

Attachments
bzip2 compressed globusrun-ws valgrind log (31.77 KB, application/octet-stream)
2006-10-09 15:33, Jim Basney 		Details
bzip2 compressed globusrun-ws valgrind log w/ globus_hashtable_lookup failure (32.28 KB, application/x-bzip)
2006-10-11 14:57, Jim Basney 		Details
globusrun-ws update package (201.99 KB, application/x-tar)
2006-10-31 15:05, Joe Bester 		Details
Updated C WS Messaging Module (253.85 KB, application/x-tar)
2006-11-02 15:24, Joe Bester 		Details
Add an attachment (proposed patch, testcase, etc.)


Note

You need to log in before you can comment on or make changes to this bug.

Description From 2006-09-26 11:37:25 
I got a core file from globusrun-ws on FC5 on x86_64.  I haven't been able to
reproduce the problem.

$ gdb globusrun-ws core.20260 
GNU gdb Red Hat Linux (6.3.0.0-1.134.fc5rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db
library "/lib64/libthread_db.so.1".

Core was generated by `globusrun-ws -submit -F dg-060.ncsa.uiuc.edu -s -c
/bin/hostname'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_c_gram_client_client_bindings_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_c_gram_client_client_bindings_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_c_gram_client_types_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_c_gram_client_types_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_delegation_client_util_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_delegation_client_util_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_c_rendezvous_client_client_bindings_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_c_rendezvous_client_client_bindings_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_c_rendezvous_client_types_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_c_rendezvous_client_types_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_c_delegation_client_client_bindings_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_c_delegation_client_client_bindings_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_c_delegation_client_types_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_c_delegation_client_types_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_notification_consumer_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_notification_consumer_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_c_rft_client_client_bindings_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_c_rft_client_client_bindings_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_c_rft_client_types_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_c_rft_client_types_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_wsrf_core_tools_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_wsrf_core_tools_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_ftp_client_gcc64dbg.so.0...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libglobus_ftp_client_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_c_wsrf_resource_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_c_wsrf_resource_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_wsa_bindings_gcc64dbg.so.0...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libglobus_wsa_bindings_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_ds_bindings_gcc64dbg.so.0...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libglobus_ds_bindings_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_wsu_bindings_gcc64dbg.so.0...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libglobus_wsu_bindings_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_wsp_bindings_gcc64dbg.so.0...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libglobus_wsp_bindings_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_wst_bindings_gcc64dbg.so.0...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libglobus_wst_bindings_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_wsse_bindings_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_wsse_bindings_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_wsseu_bindings_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_wsseu_bindings_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_wsc_bindings_gcc64dbg.so.0...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libglobus_wsc_bindings_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_wsbf_bindings_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_wsbf_bindings_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_wsnt_bindings_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_wsnt_bindings_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_wsrl_bindings_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_wsrl_bindings_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_wsrp_bindings_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_wsrp_bindings_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_wssg_bindings_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_wssg_bindings_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_notification_consumer_bindings_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_notification_consumer_bindings_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_subscription_manager_bindings_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_subscription_manager_bindings_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_ftp_control_gcc64dbg.so.0...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libglobus_ftp_control_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_ws_messaging_gcc64dbg.so.0...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libglobus_ws_messaging_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_usage_gcc64dbg.so.0...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libglobus_usage_gcc64dbg.so.0
Reading symbols from /usr/local/gt-4.0.3/lib/libglobus_io_gcc64dbg.so.0...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libglobus_io_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_xio_gcc64dbg.so.0...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libglobus_xio_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libgssapi_error_gcc64dbg.so.0...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libgssapi_error_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_gss_assist_gcc64dbg.so.0...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libglobus_gss_assist_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_gssapi_gsi_gcc64dbg.so.0...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libglobus_gssapi_gsi_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_gsi_proxy_core_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_gsi_proxy_core_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_ws_security_gcc64dbg.so.0...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libglobus_ws_security_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_gsi_credential_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_gsi_credential_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_gsi_callback_gcc64dbg.so.0...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libglobus_gsi_callback_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_oldgaa_gcc64dbg.so.0...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libglobus_oldgaa_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_gsi_sysconfig_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_gsi_sysconfig_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_gsi_cert_utils_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_gsi_cert_utils_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_openssl_gcc64dbg.so.0...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libglobus_openssl_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_openssl_error_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_openssl_error_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_callout_gcc64dbg.so.0...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libglobus_callout_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_proxy_ssl_gcc64dbg.so.0...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libglobus_proxy_ssl_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_common_gcc64dbg.so.0...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libglobus_common_gcc64dbg.so.0
Reading symbols from /usr/local/gt-4.0.3/lib/libssl_gcc64dbg.so.0...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libssl_gcc64dbg.so.0
Reading symbols from /usr/local/gt-4.0.3/lib/libcrypto_gcc64dbg.so.0...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libcrypto_gcc64dbg.so.0
Reading symbols from /usr/local/gt-4.0.3/lib/libxml2_gcc64dbg.so.2...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libxml2_gcc64dbg.so.2
Reading symbols from /usr/local/gt-4.0.3/lib/libltdl_gcc64dbg.so.3...done.
Loaded symbols for /usr/local/gt-4.0.3/lib/libltdl_gcc64dbg.so.3
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_xio_buffer_driver_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_xio_buffer_driver_gcc64dbg.so.0
Reading symbols from /lib64/libnss_files.so.2...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /lib64/libnss_dns.so.2...done.
Loaded symbols for /lib64/libnss_dns.so.2
Reading symbols from /lib64/libresolv.so.2...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_handler_ws_secure_message_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_handler_ws_secure_message_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_handler_ws_addressing_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_handler_ws_addressing_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/globus_service_modules/wsrf/services/libNotificationConsumerService_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/globus_service_modules/wsrf/services/libNotificationConsumerService_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_wsrp_GetResourceProperty_provider_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_wsrp_GetResourceProperty_provider_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_wsrp_GetMultipleResourceProperties_provider_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_wsrp_GetMultipleResourceProperties_provider_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_wsrp_SetResourceProperties_provider_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_wsrp_SetResourceProperties_provider_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_wsrp_QueryResourceProperties_provider_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_wsrp_QueryResourceProperties_provider_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_wsrl_ImmediateResourceTermination_provider_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_wsrl_ImmediateResourceTermination_provider_gcc64dbg.so.0
Reading symbols from
/usr/local/gt-4.0.3/lib/libglobus_wsrl_ScheduledResourceTermination_provider_gcc64dbg.so.0...done.
Loaded symbols for
/usr/local/gt-4.0.3/lib/libglobus_wsrl_ScheduledResourceTermination_provider_gcc64dbg.so.0
#0  0x00000000000008c0 in ?? ()
(gdb) bt
#0  0x00000000000008c0 in ?? ()
#1  0x00002aaaae9f4b04 in globus_hashtable_lookup (table=0x570fa0,
key=0x581230) at globus_hashtable.c:390
#2  0x00002aaaad5e2bd2 in globus_xsd_type_registry_get (registry=0x570fa0,
name=0x581230, info=0x7fff67d62800) at globus_xsd_type_registry.c:348
#3  0x00002aaaad65c4dc in globus_i_xsd_deserialize_from_element_qname
(deselem_qname=0x581230, found_any_info=0x7fff67d629b8, registry=0x0,
message_handle=0x5ec4a0) at xsd_any.c:259
#4  0x00002aaaad65d4de in xsd_any_deserialize (element_qname=0x7fff67d62b60,
instance=0x7fff67d629b0, message_handle=0x5ec4a0, options=0) at xsd_any.c:561
#5  0x00002aaaad65b0dc in xsd_any_array_deserialize
(element_qname=0x7fff67d62b60, array=0x579ed0, message=0x5ec4a0, options=0) at
xsd_any.c:26
#6  0x00002aaaabe9ed7e in wsa_ReferencePropertiesType_deserialize_pointer
(element_qname=0x7fff67d62d70, ip=0x5b72d0, message_handle=0x5ec4a0, options=0)
at wsa_ReferencePropertiesType.c:979
#7  0x00002aaaabe975c4 in wsa_EndpointReferenceType_deserialize
(element_qname=0x2aaaabfc1bf0, instance=0x5b72c0, message_handle=0x5ec4a0,
options=0) at wsa_EndpointReferenceType.c:1101
#8  0x00002aaaaf6c1483 in globus_l_handler_ws_addressing_client_invoke_From
(session=0x5a0540, message_handle=0x5ec4a0, result=0) at
globus_handler_ws_addressing.c:787
#9  0x00002aaaad5d98d0 in globus_l_handler_callback_call_next (args=0x5a0540)
at globus_handler_chain.c:677
#10 0x00002aaaae9ec624 in globus_callback_space_poll (timestop=0x525e60,
space=-2) at globus_callback_nothreads.c:1430
#11 0x00002aaaabc3fbe4 in globus_l_ftp_client_deactivate () at
globus_ftp_client.c:181
#12 0x00002aaaae9fa5a1 in globus_module_deactivate (module_descriptor=0x525530)
at globus_module.c:311
#13 0x00002aaaae9fa677 in globus_module_deactivate_all () at
globus_module.c:369
#14 0x0000000000415f78 in main (argc=7, argv=0x7fff67d63978) at
globusrun_ws.c:775
(gdb) up
#1  0x00002aaaae9f4b04 in globus_hashtable_lookup (table=0x570fa0,
key=0x581230) at globus_hashtable.c:390
390     globus_hashtable.c: No such file or directory.
        in globus_hashtable.c
(gdb) p *itable
$1 = {size = 5717776, load = 0, buckets = 0x21, first = 0x37d4943a00, last =
0x576050, current = 0x20, hash_func = 0x8c0, keyeq_func = 0xffffffff9ea62c80,
memory = 0xffffffff9fbaf970}
(gdb) quit
------- Comment #1 From 2006-09-27 10:20:45 ------- 
I got two more core files last night.  Looks like the same problem.
In case it's helpful, here's the info from those core files:

$ gdb globusrun-ws core.26347 
GNU gdb Red Hat Linux (6.3.0.0-1.134.fc5rh)
[...]
This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db
library "/lib64/libthread_db.so.1".

Core was generated by `globusrun-ws -submit -F dg-034.ncsa.uiuc.edu -s -c
/bin/hostname'.
Program terminated with signal 11, Segmentation fault.
[...]
(gdb) bt
#0  0x00000000000008c0 in ?? ()
#1  0x00002aaaae9f4b04 in globus_hashtable_lookup (table=0x570fa0,
key=0x543140) at globus_hashtable.c:390
#2  0x00002aaaad5e2bd2 in globus_xsd_type_registry_get (registry=0x570fa0,
name=0x543140, info=0x7fff77b89630) at globus_xsd_type_registry.c:348
#3  0x00002aaaad65c4dc in globus_i_xsd_deserialize_from_element_qname
(deselem_qname=0x543140, found_any_info=0x7fff77b897e8, registry=0x0,
message_handle=0x610d30) at xsd_any.c:259
#4  0x00002aaaad65d4de in xsd_any_deserialize (element_qname=0x7fff77b89990,
instance=0x7fff77b897e0, message_handle=0x610d30, options=0) at xsd_any.c:561
#5  0x00002aaaad65b0dc in xsd_any_array_deserialize
(element_qname=0x7fff77b89990, array=0x60e0a0, message=0x610d30, options=0) at
xsd_any.c:26
#6  0x00002aaaabe9ed7e in wsa_ReferencePropertiesType_deserialize_pointer
(element_qname=0x7fff77b89ba0, ip=0x578dd0, message_handle=0x610d30, options=0)
at wsa_ReferencePropertiesType.c:979
#7  0x00002aaaabe975c4 in wsa_EndpointReferenceType_deserialize
(element_qname=0x2aaaabfc1bf0, instance=0x578dc0, message_handle=0x610d30,
options=0) at wsa_EndpointReferenceType.c:1101
#8  0x00002aaaaf6c1483 in globus_l_handler_ws_addressing_client_invoke_From
(session=0x60e060, message_handle=0x610d30, result=0) at
globus_handler_ws_addressing.c:787
#9  0x00002aaaad5d98d0 in globus_l_handler_callback_call_next (args=0x60e060)
at globus_handler_chain.c:677
#10 0x00002aaaae9ec624 in globus_callback_space_poll (timestop=0x525e60,
space=-2) at globus_callback_nothreads.c:1430
#11 0x00002aaaabc3fbe4 in globus_l_ftp_client_deactivate () at
globus_ftp_client.c:181
#12 0x00002aaaae9fa5a1 in globus_module_deactivate (module_descriptor=0x525530)
at globus_module.c:311
#13 0x00002aaaae9fa677 in globus_module_deactivate_all () at
globus_module.c:369
#14 0x0000000000415f78 in main (argc=7, argv=0x7fff77b8a7a8) at
globusrun_ws.c:775
(gdb) up
#1  0x00002aaaae9f4b04 in globus_hashtable_lookup (table=0x570fa0,
key=0x543140) at globus_hashtable.c:390
390     globus_hashtable.c: No such file or directory.
        in globus_hashtable.c
(gdb) p *itable
$1 = {size = 5717776, load = 0, buckets = 0x21, first = 0x552600, last =
0x576050, current = 0x20, hash_func = 0x8c0, keyeq_func = 0xffffffff9ea62c80,
memory = 0xffffffff9fbaf970}
(gdb) up
#2  0x00002aaaad5e2bd2 in globus_xsd_type_registry_get (registry=0x570fa0,
name=0x543140, info=0x7fff77b89630) at globus_xsd_type_registry.c:348
348     globus_xsd_type_registry.c: No such file or directory.
        in globus_xsd_type_registry.c
(gdb) p *name
$3 = {Namespace = 0x60d2b0 "http://www.globus.org/namespaces/2004/06/core",
local = 0x578a70 "SubscriptionKey"}

$ gdb globusrun-ws core.26157 
GNU gdb Red Hat Linux (6.3.0.0-1.134.fc5rh)
[...]
This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db
library "/lib64/libthread_db.so.1".

Core was generated by `globusrun-ws -submit -F dg-062.ncsa.uiuc.edu -s -c
/bin/hostname'.
Program terminated with signal 11, Segmentation fault.
[...]
(gdb) bt
#0  0x00000000000008c0 in ?? ()
#1  0x00002aaaae9f4b04 in globus_hashtable_lookup (table=0x570fa0,
key=0x543140) at globus_hashtable.c:390
#2  0x00002aaaad5e2bd2 in globus_xsd_type_registry_get (registry=0x570fa0,
name=0x543140, info=0x7fff7a155c00) at globus_xsd_type_registry.c:348
#3  0x00002aaaad65c4dc in globus_i_xsd_deserialize_from_element_qname
(deselem_qname=0x543140, found_any_info=0x7fff7a155db8, registry=0x0,
message_handle=0x61f6f0) at xsd_any.c:259
#4  0x00002aaaad65d4de in xsd_any_deserialize (element_qname=0x7fff7a155f60,
instance=0x7fff7a155db0, message_handle=0x61f6f0, options=0) at xsd_any.c:561
#5  0x00002aaaad65b0dc in xsd_any_array_deserialize
(element_qname=0x7fff7a155f60, array=0x5d7650, message=0x61f6f0, options=0) at
xsd_any.c:26
#6  0x00002aaaabe9ed7e in wsa_ReferencePropertiesType_deserialize_pointer
(element_qname=0x7fff7a156170, ip=0x578dd0, message_handle=0x61f6f0, options=0)
at wsa_ReferencePropertiesType.c:979
#7  0x00002aaaabe975c4 in wsa_EndpointReferenceType_deserialize
(element_qname=0x2aaaabfc1bf0, instance=0x578dc0, message_handle=0x61f6f0,
options=0) at wsa_EndpointReferenceType.c:1101
#8  0x00002aaaaf6c1483 in globus_l_handler_ws_addressing_client_invoke_From
(session=0x5d7610, message_handle=0x61f6f0, result=0) at
globus_handler_ws_addressing.c:787
#9  0x00002aaaad5d98d0 in globus_l_handler_callback_call_next (args=0x5d7610)
at globus_handler_chain.c:677
#10 0x00002aaaae9ec624 in globus_callback_space_poll (timestop=0x525e60,
space=-2) at globus_callback_nothreads.c:1430
#11 0x00002aaaabc3fbe4 in globus_l_ftp_client_deactivate () at
globus_ftp_client.c:181
#12 0x00002aaaae9fa5a1 in globus_module_deactivate (module_descriptor=0x525530)
at globus_module.c:311
#13 0x00002aaaae9fa677 in globus_module_deactivate_all () at
globus_module.c:369
#14 0x0000000000415f78 in main (argc=7, argv=0x7fff7a156d78) at
globusrun_ws.c:775
(gdb) up
#1  0x00002aaaae9f4b04 in globus_hashtable_lookup (table=0x570fa0,
key=0x543140) at globus_hashtable.c:390
390     globus_hashtable.c: No such file or directory.
        in globus_hashtable.c
(gdb) p *itable
$1 = {size = 5717776, load = 0, buckets = 0x21, first = 0x3b6ff43a00, last =
0x576050, current = 0x20, hash_func = 0x8c0, keyeq_func = 0xffffffff9ea62c80,
memory = 0xffffffff9fbaf970}
(gdb) up
#2  0x00002aaaad5e2bd2 in globus_xsd_type_registry_get (registry=0x570fa0,
name=0x543140, info=0x7fff7a155c00) at globus_xsd_type_registry.c:348
348     globus_xsd_type_registry.c: No such file or directory.
        in globus_xsd_type_registry.c
(gdb) p *name
$2 = {Namespace = 0x605420 "http://www.globus.org/namespaces/2004/06/core",
local = 0x578a70 "SubscriptionKey"}

I'm running globusrun-ws once per hour across 100 machines and got 3
segmentation faults in a 24 hour period, so that's a 0.125% failure
rate.  The segmentation fault is happening very rarely.

I looked around in the code and didn't see any obvious bugs.
------- Comment #2 From 2006-10-09 14:04:44 ------- 
It's hard to tell from this stack trace what is failing. Do you have info about
the state of the job at the time of destruction? Did it complete with FAILED or
DONE, did you get any streamed output, etc. If you have a tool like valgrind
you could run globusrun-ws with, it would help locating this problem.

joe
------- Comment #3 From 2006-10-09 14:17:41 ------- 
Here's an example of the command output:

$ globusrun-ws -submit -F $HOSTNAME -s -c /bin/hostname
Delegating user credentials...Done.
Submitting job...Done.
Job ID: uuid:99c67386-4d3f-11db-b1f8-00163e989860
Termination time: 09/27/2006 09:15 GMT
Current job state: Active
Current job state: CleanUp-Hold
dg-060.ncsa.uiuc.edu
Current job state: CleanUp
Current job state: Done
Destroying job...Done.
Cleaning up any delegated credentials...Done.
Segmentation fault      (core dumped)

It looks like the segmentation fault always happens after the "Cleaning up any
delegated credentials...Done." message.  I'll work on getting valgrind output
for you.
------- Comment #4 From 2006-10-09 15:33:16 ------- 
Created an attachment (id=1075) [details]
bzip2 compressed globusrun-ws valgrind log

The segmentation fault didn't occur in this run.
------- Comment #5 From 2006-10-11 14:57:07 ------- 
Created an attachment (id=1077) [details]
bzip2 compressed globusrun-ws valgrind log w/ globus_hashtable_lookup failure

Here's a valgrind log showing an error in globus_hashtable_lookup at the end. 
This run had the following output.

$ valgrind --error-limit=no --log-file-exactly=/tmp/valgrind.log.$$
globusrun-ws -submit -F $HOSTNAME -s -c /bin/hostname
Delegating user credentials...Done.
Submitting job...Done.
Job ID: uuid:ceaf4212-594c-11db-9f69-00163e989809
Termination time: 10/12/2006 17:20 GMT
Current job state: Active
Current job state: CleanUp-Hold
dg-009.ncsa.uiuc.edu
Current job state: CleanUp
Current job state: Done
Destroying job...Done.
Cleaning up any delegated credentials...Done.
Assertion 0 && "globus_hashtable_lookup bad parms" failed in file
globus_hashtable.c at line 385
Aborted
------- Comment #6 From 2006-10-31 15:05:09 ------- 
Created an attachment (id=1115) [details]
globusrun-ws update package

Could you try installing this update package to replace your client? I've not
been able to reproduce the problem, but based on the stack traces and valgrind
reports I have a theory about what's going on. This patch adds a wait for
callbacks to occur before deactivating the various library modules used by
globusrun-ws. I think the callback for the Destroy operation is outstanding in
the crash case you reported.

There are instructions for installing update packages on the advisories web
page: http://www.globus.org/toolkit/advisories.html

joe
------- Comment #7 From 2006-11-01 13:20:56 ------- 
I'm sorry to report that I still get the segmentation fault after applying the
update.

$ globusrun-ws -version
globusrun-ws: 0.14
$ globusrun-ws -submit -F $HOSTNAME -s -c /bin/hostname
Delegating user credentials...Done.
Submitting job...Done.
Job ID: uuid:34f3f6d2-69db-11db-b14a-00163e989805
Termination time: 11/02/2006 19:00 GMT
Current job state: Active
Current job state: CleanUp-Hold
dg-005.ncsa.uiuc.edu
Current job state: CleanUp
Current job state: Done
Destroying job...Done.
Cleaning up any delegated credentials...Done.
Segmentation fault      (core dumped) 
$ gdb globusrun-ws core.21578 
GNU gdb Red Hat Linux (6.3.0.0-1.134.fc5rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db
library "/lib64/libthread_db.so.1".

Core was generated by `globusrun-ws -submit -F dg-005.ncsa.uiuc.edu -s -c
/bin/hostname'.
Program terminated with signal 11, Segmentation fault.
[...]
(gdb) bt
#0  0x00000000000008c0 in ?? ()
#1  0x00002aaaae9f4b04 in globus_hashtable_lookup (table=0x570fa0,
key=0x543140) at globus_hashtable.c:390
#2  0x00002aaaad5e2bd2 in globus_xsd_type_registry_get (registry=0x570fa0,
name=0x543140, info=0x7fff44411eb0) at globus_xsd_type_registry.c:348
#3  0x00002aaaad65c4dc in globus_i_xsd_deserialize_from_element_qname
(deselem_qname=0x543140, found_any_info=0x7fff44412068, registry=0x0,
message_handle=0x61f700) at xsd_any.c:259
#4  0x00002aaaad65d4de in xsd_any_deserialize (element_qname=0x7fff44412210,
instance=0x7fff44412060, message_handle=0x61f700, options=0) at xsd_any.c:561
#5  0x00002aaaad65b0dc in xsd_any_array_deserialize
(element_qname=0x7fff44412210, array=0x60cea0, message=0x61f700, options=0) at
xsd_any.c:26
#6  0x00002aaaabe9ed7e in wsa_ReferencePropertiesType_deserialize_pointer
(element_qname=0x7fff44412420, ip=0x578dd0, message_handle=0x61f700, options=0)
at wsa_ReferencePropertiesType.c:979
#7  0x00002aaaabe975c4 in wsa_EndpointReferenceType_deserialize
(element_qname=0x2aaaabfc1bf0, instance=0x578dc0, message_handle=0x61f700,
options=0) at wsa_EndpointReferenceType.c:1101
#8  0x00002aaaaf6c1483 in globus_l_handler_ws_addressing_client_invoke_From
(session=0x60ce60, message_handle=0x61f700, result=0) at
globus_handler_ws_addressing.c:787
#9  0x00002aaaad5d98d0 in globus_l_handler_callback_call_next (args=0x60ce60)
at globus_handler_chain.c:677
#10 0x00002aaaae9ec624 in globus_callback_space_poll (timestop=0x525f00,
space=-2) at globus_callback_nothreads.c:1430
#11 0x00002aaaabc3fbe4 in globus_l_ftp_client_deactivate () at
globus_ftp_client.c:181
#12 0x00002aaaae9fa5a1 in globus_module_deactivate (module_descriptor=0x5255d0)
at globus_module.c:311
#13 0x00002aaaae9fa677 in globus_module_deactivate_all () at
globus_module.c:369
#14 0x0000000000416004 in main (argc=7, argv=0x7fff44413028) at
globusrun_ws.c:775
------- Comment #8 From 2006-11-02 15:24:01 ------- 
Created an attachment (id=1117) [details]
Updated C WS Messaging Module

I noticed a kind of strange thing while running in a debugger (a count of
outstanding callbacks was going negative). I've attached an updated version of
the globus_c_ws_messaging module which causes the "service engine done"
callback to occur only when the final outstanding session has terminated.
Hopefully this, combined with the other, will fix the problem.

joe
------- Comment #9 From 2006-11-05 22:27:31 ------- 
I think the two update packages have solved the problem.  I'm not seeing the
segmentation fault anymore.  Thanks!
------- Comment #10 From 2006-11-06 10:36:43 ------- 
Committed to CVS. I Put update packages on the ftp site and added an advisory
to http://www.globus.org/toolkit/advisories.html containing the packages which
fix the problem.

joe
First Last Prev Next    No search results available