Bugzilla – Bug 6415
GSSAPI symbol clash with kerberos causes problems when dlopening GSI libraries
Last modified: 2008-10-22 10:57:29
You need to log in before you can comment on or make changes to this bug.
The GSSAPI library is deliberatly using the same symbols as other GSSAPI implementations, so that it can be used in a system that loads different GSSAPI implementations dynamically at runtime. However, if a library or application links to the GSSAPI library at compilation time this causes problems, since the symbols of the API can be resolved to use any GSSAPI implementation if more than one of the GSSAPI libraries are loaded in the applications address space. In the case of ARC0 this is usually not a problem, since you can make sure the globus GSSAPI implementations is loaded first by putting it first in the linking order of the application. However, in the ARC1 case the globus GSSAPI is only used in the plugins, so if the main applications has already loaded a different GSSAPI implementation this will be used even if the plugin is linked to the globus GSSAPI library: [ellert@ellert arc1]$ ARC_PLUGIN_PATH=src/hed/dmc/file/.libs:src/hed/dmc/gridftp/.libs src/clients/cream2/glitecp std.in gsiftp://grid.tsl.uu.se/storage/ellert/fil -d FATAL /tmp/cream_client_test//std.in -> gsiftp://grid.tsl.uu.se/storage/ellert/fil 0 s: 0,0 kB 0,0 kB/s 0,0 kB/s . . . 0 s: 0,0 kB 0,0 kB/s 0,0 kB/s . . . 0 s: 0,0 kB 0,0 kB/s 0,0 kB/s . . . 0 s: 0,0 kB 0,0 kB/s 0,0 kB/s . . . 0 s: 0,0 kB 0,0 kB/s 0,0 kB/s . . . File moving was not succeeded. With some more verbose debug (to talkative to quote here in its entirety) you get the error message: "Unspecified GSS failure. No credentials cache found". The No credentials cache found error is a kerberos error message from the kerberos GSSAPI implementation. You can work around the problem bu LD_PRELOADing the globus GSSAPI implementation: [ellert@ellert arc1]$ LD_LIBRARY_PATH=../../globus-dbg/lib LD_PRELOAD=../../globus-dbg/lib/libglobus_gssapi_gsi_gcc32dbgpthr.so.0 ARC_PLUGIN_PATH=src/hed/dmc/file/.libs:src/hed/dmc/gridftp/.libs src/clients/cream2/glitecp std.in gsiftp://grid.tsl.uu.se/storage/ellert/fil -d FATAL /tmp/cream_client_test//std.in -> gsiftp://grid.tsl.uu.se/storage/ellert/fil 0 s: 0,0 kB 0,0 kB/s 0,0 kB/s . . . However, this is not really a workable solution. The real solution is to add version tags to the GSSAPI symbols in the globus GSSAPI library. There are at least 2 other GSSAPI implementations in the Fedora linux distribution and both of these have version tags on their symbols: [ellert@ellert test]$ cat gssapitest.c #include <gssapi.h> int main() { gss_release_cred(NULL, NULL); } [ellert@ellert test]$ gcc -o gssapitest gssapitest.c -lgssapi_krb5 [ellert@ellert test]$ nm gssapitest | grep gss_release_cred U gss_release_cred@@gssapi_krb5_2_MIT [ellert@ellert test]$ gcc -o gssapitest gssapitest.c -lgssglue [ellert@ellert test]$ nm gssapitest | grep gss_release_cred U gss_release_cred@@libgssapi_CITI_2 Whereas the globus version does not have it: [ellert@ellert test]$ LD_LIBRARY_PATH=../globus-dbg/lib gcc -o gssapitest gssapitest.c -L ../globus-dbg/lib -lglobus_gssapi_gsi_gcc32dbgpthr [ellert@ellert test]$ nm gssapitest | grep gss_release_cred U gss_release_cred The attached patch adds a version tag to the GSSAPI symbols in the globus GSSAPI library: [ellert@ellert test]$ LD_LIBRARY_PATH=../globus-dbg2/lib gcc -o gssapitest gssapitest.c -L ../globus-dbg2/lib -lglobus_gssapi_gsi_gcc32dbgpthr [ellert@ellert test]$ nm gssapitest | grep gss_release_cred U gss_release_cred@@globus_gssapi_gsi When the rest of globus and ARC1 are compiled against this patched version of globus GSSAPI it works without the LD_PREOPEN workaround: [ellert@ellert arc1]$ ARC_PLUGIN_PATH=src/hed/dmc/file/.libs:src/hed/dmc/gridftp/.libs src/clients/cream2/glitecp std.in gsiftp://grid.tsl.uu.se/storage/ellert/fil -d FATAL /tmp/cream_client_test//std.in -> gsiftp://grid.tsl.uu.se/storage/ellert/fil 0 s: 0,0 kB 0,0 kB/s 0,0 kB/s . . . Since it is not possible to attach patches to the Globus bugzilla I refer to the attachments of the original NorduGrid bugzilla report for the patch.
This patch is linker specific (fails to link on machines without GNU ld symbol versioning), so I'm reluctant to add it to the GT distribution. Can you tell me more about how this will affect things: - What versions of the linker support this option? - Will applications linked against one GSSAPI be able to substitute, at runtime, a different GSSAPI? Also, have you looked at http://grid.ncsa.uiuc.edu/gssapi-mechglue/ from NCSA which allows an application to link in one GSSAPI which delegates to other GSSAPI implementations? Part of the advantage of using GSSAPI is that an application isn't bound to any particular security implementation. It's unfortunate that is causing problems for you in this case.
The other potential solution would be to link your plugin with the static gssapi library that it requires.
(In reply to comment #1) > This patch is linker specific (fails to link on machines without GNU ld symbol > versioning), so I'm reluctant to add it to the GT distribution. Can you tell me > more about how this will affect things: > > - What versions of the linker support this option? > - Will applications linked against one GSSAPI be able to substitute, at > runtime, a different GSSAPI? There are two ways to use a GSSAPI library. Either you develop against a specific version of the GSSAPI library using knowledge of specific details in its implementation and link directly to that specific version. Or your code is portable enough to use any version of the GSSAPI library and you use dlopen to load the desired GSSAPI version at runtime (or use libgssglue for the same effect). The use case you describe - to explicitly link against one version of the GSSAPI library and at runtime use a different one is broken, and neither the kerberos nor the libgssglue GSSAPI implementation supports this (since they both have versioned symbols). > Also, have you looked at http://grid.ncsa.uiuc.edu/gssapi-mechglue/ from NCSA > which allows an application to link in one GSSAPI which delegates to other > GSSAPI implementations? Yes I have seen it. Using the libgssglue is an alternative to using dlopen if your code is portable enough to use any version of the GSSAPI library. The libgssglue itself does have versioned symbols to make sure that if you link against it you will also get this GSSAPI version. > Part of the advantage of using GSSAPI is that an application isn't bound to any > particular security implementation. It's unfortunate that is causing problems > for you in this case. In this case you should use dlopen or libgssglue and not link against one GSSAPI library explicitly and expect to be able to use a different one at runtime. (You can possibly not link against any GSSAPI library at all if you are allowed to have unresolved symbols at linktime. Not all platforms support this.) (In reply to comment #2) > The other potential solution would be to link your plugin with the static > gssapi library that it requires. This does not really work either. There is an established solution to this problem used by both the kerberos and the libgssglue GSSAPI libraries. Why would globus try to invent a different solution, when the established solution works. The libgssglue source uses -Wl,--version-script always. The kerberos source uses it for systems matching *-*-linux* | *-*-gnu* | *-*-k*bsd*-gnu) If you want I can implement such a restriction in the globus patch.
My resistance to this patch is purely practical. The one you proposed doesn't compile on all of the systems we support. It's unfortunate that you see that as Globus being against some established solution because of that. My other comments were meant to make sure that you'd exhausted other more-portable alternatives. If you have a patch that does work, we would consider it.
Hi! Will this patch work for you? http://www.grid.tsl.uu.se/repos/globus/fedora/9/info/globus-gssapi-gsi-labels.patch It uses the same condition for when to use the labelled symbols as the kerberos gssapi sources.
I tried that new patch on a debian linux machine and it didn't enable the versioning like I'd expected.
On my Fedora 9 machine it works. The build log says "-Wl,--version-script -Wl,gssapi.sym" on the lines where the linking of the library is done. Also this small check shows that it works: [ellert@ellert ~]$ cat test.c void gss_acquire_cred(); int main() { gss_acquire_cred(); return 0; } [ellert@ellert ~]$ gcc -o test test.c -lglobus_gssapi_gsi [ellert@ellert ~]$ nm test | grep gss_acquire_cred U gss_acquire_cred@@globus_gssapi_gsi Did you rebootstrap? This version of the patch changes configure.in and Makefile.am without changing configure and Makefile.in. I could provide you with a patch that includes the propagated changes to configure and Makefile.in if you prefer. What is the output of config.guess on your debian machine. On my Fedora 9 it is "i686-pc-linux-gnu", which matches the test I introduced in the patch.
I did a little snooping. It looks like $target_alias is undefined in my build unless I explicitly pass a --target to the configure script, which is not the default for gpt builds.
OK, now I get it. I have updated the patch at the same URL. Now it uses $GLOBUS_HOST that is picked up from globus-build-env-${flavor}.sh instead - this should work for a pure GPT build as well.
Committed the latest patch plus another modification to Makefile.am to add gssapi.sym to EXTRA_DIST so that it can be built from a source package.