Bug 2413 - Decrypt Error
: Decrypt Error
Status: NEW
: CoG jglobus
security
: unspecified
: PC Linux
: P2 critical
: ---
Assigned To:
:
:
:
:
  Show dependency treegraph
 
Reported: 2004-12-07 21:41 by
Modified: 2007-09-12 16:29 (History)


Attachments


Note

You need to log in before you can comment on or make changes to this bug.


Description From 2004-12-07 21:41:12
I just updated the jglobus jar to the latest, built from the cvs, and
tried to run an application, which at one point delegates the user
credetial to another component of our distributed storage system.

The code that worked fine before started to fail with an exception:

12/07 13:22:44 Cell(srm_1@fnisd1Domain) :
org.globus.common.ChainedIOException: Authentication failed [Caused by:
Failure unspecified at GSS-API level (Mech
anism level: Cannot request delegation without authorization (target
name null))]. Caused by GSSException: Failure unspecified at GSS-API
level (Mechanism l
evel: Cannot request delegation without authorization (target name null))
12/07 13:22:44 Cell(srm_1@fnisd1Domain) :       at
org.globus.gsi.gssapi.GlobusGSSContextImpl.initSecContext(GlobusGSSContextImpl.java:462)
12/07 13:22:44 Cell(srm_1@fnisd1Domain) :       at
org.globus.gsi.gssapi.net.GssSocket.authenticateClient(GssSocket.java:97)
12/07 13:22:44 Cell(srm_1@fnisd1Domain) :       at
org.globus.gsi.gssapi.net.GssSocket.startHandshake(GssSocket.java:135)
12/07 13:22:44 Cell(srm_1@fnisd1Domain) :       at
diskCacheV111.srm.security.SslGsiSocketFactory.delegateCredential(SslGsiSocketFactory.java:1037)
12/07 13:22:44 Cell(srm_1@fnisd1Domain) :       at
diskCacheV111.srm.dcache.Storage.delegate(Storage.java:2803)
12/07 13:22:44 Cell(srm_1@fnisd1Domain) :       at
diskCacheV111.srm.dcache.Storage.access$700(Storage.java:237)
12/07 13:22:44 Cell(srm_1@fnisd1Domain) :       at
diskCacheV111.srm.dcache.Storage$3.run(Storage.java:2756)
12/07 13:22:44 Cell(srm_1@fnisd1Domain) :       at
java.lang.Thread.run(Thread.java:534)


Here is the listing of the function that does the delegation, it is
clear from the code that I do specify the HostAuthorization as the
authorization mechanizm:

   public static void  delegateCredential(java.net.InetAddress inetAddress,
    int port,GSSCredential credential,boolean fulldelegation)
    throws Exception {
        // say("createSocket("+inetAddress+","+port+")");
        Socket s =null;
        try {
            //   say("delegateCredentials() user credential is
"+credential);
            GSSManager manager = ExtendedGSSManager.getInstance();
            ExtendedGSSContext context =
            (ExtendedGSSContext) manager.createContext(credential);
            context.setOption(GSSConstants.GSS_MODE,
            GSIConstants.MODE_GSI);
            context.requestCredDeleg(true);
            if(fulldelegation) {
                context.setOption(GSSConstants.DELEGATION_TYPE,
GSIConstants.DELEGATION_TYPE_FULL);
            }
            else {
                context.setOption(GSSConstants.DELEGATION_TYPE,
GSIConstants.DELEGATION_TYPE_LIMITED);
            }
            s = new Socket(inetAddress,port);
            GSIGssSocket gsiSocket = new GSIGssSocket(s, context);
            gsiSocket.setUseClientMode(true);
            gsiSocket.setAuthorization(
            org.globus.gsi.gssapi.auth.HostAuthorization.getInstance());
            gsiSocket.setWrapMode(GssSocket.SSL_MODE);
            gsiSocket.startHandshake();
        }
        catch(Exception e) {
            if(s!=null) {
                try {
                    s.close();
                }
                catch(Exception e1) {
                }
            }
            throw e;
        }
    }



Thanks a lot for your help
Timur Perelmutov
Fermilab
------- Comment #1 From 2004-12-09 14:55:10 -------
I figured out why the "Authentication failed" exception was happening. The main
reason was that I was creating the GSSContext with this command: 
ExtendedGSSContext context =
            (ExtendedGSSContext) manager.createContext(credential);

When in fact I was supposed to use this one:            

(ExtendedGSSContext) manager.createContext(peer,
                GSSConstants.MECH_OID,
                credential,
                GSSContext.DEFAULT_LIFETIME);

It has nothing to do with the GSI Socket Authentication, it is an internal peer
name authentication step, that was not present in the earlier releases, that was
causing this problem. 

So now we observe a different behaviour, the delegation step works with most of
the peers most of the time, but then suddenly it starts to fail with a
mestirious "Decrypt Error" failure on the receiving side. This behaviour
continues for some time, then it goes away. We obesrved it with the previos
version of jglobus libraries (built out of cvs last spring), and we were hoping
to see this behavior gone with the updated jglobus. But we still see this. We
really hope you will help us, because it is a real show stopper for us. Here are
the details. 
1. The credential is delegated from the client to the server for performance of
the gridftp transfers by the server on behalf of the client. We never observed
any problems on this step of delegation, we perform full delegation in this step. 
2. The delegated credential is delegated from the server to one of the multiple
disk nodes, where it is to be utilized for performing the gridftp transfer. Once
the credntial is delegated to the disk node, we never see any problems with
gridftp transfer step. But unfortunately the step of second delegation is the
one that fails sometimes.
Here is the exception we see on the disk node, receiving the credential:

org.globus.common.ChainedIOException: Authentication failed [Caused by: Failure
unspecified at GSS-API level [Root error message: Decrypt error]]. Caused by
GSSException: Failure unspecified at GSS-API level [Root error message: Decrypt
error].  Root exception is COM.claymoresystems.ptls.SSLThrewAlertException:
Decrypt error
     at COM.claymoresystems.ptls.SSLConn.alert(SSLConn.java:235)
     at
COM.claymoresystems.ptls.SSLCertificateVerify.decode(SSLCertificateVerify.java:160)
     at
COM.claymoresystems.ptls.SSLHandshakeServer.recvCertificateVerify(SSLHandshakeServer.java:561)
     at
COM.claymoresystems.ptls.SSLHandshakeServer.processTokens(SSLHandshakeServer.java:228)
     at
COM.claymoresystems.ptls.SSLHandshake.processHandshake(SSLHandshake.java:135)
     at
org.globus.gsi.gssapi.GlobusGSSContextImpl.acceptSecContext(GlobusGSSContextImpl.java:276)
     at org.globus.gsi.gssapi.net.GssSocket.authenticateServer(GssSocket.java:119)
     at org.globus.gsi.gssapi.net.GssSocket.startHandshake(GssSocket.java:137)
     at
diskCacheV111.movers.RemoteGsiftpTransferProtocol_1.runIO(RemoteGsiftpTransferProtocol_1.java:265)
     at
diskCacheV111.pools.MultiProtocolPool2$RepositoryIoHandler.run(MultiProtocolPool2.java:1154)
     at diskCacheV111.util.SimpleJobScheduler$SJob.run(SimpleJobScheduler.java:63)
     at java.lang.Thread.run(Thread.java:534)


Here is the code that is used to receive the credential:

        Socket deleg_socket = ss.accept();
        GSSCredential deleg_cred ;
        try
        {
            GSSContext context = getServerContext();
            GSIGssSocket gsiSocket = new GSIGssSocket(deleg_socket, context);
            gsiSocket.setUseClientMode(false);
            gsiSocket.setAuthorization(
            new Authorization() {
                public void authorize(GSSContext context, String host)
                throws AuthorizationException {
                    //we might add some authorization here later
                    //but in general we trust that the connection
                    //came from a head node and user was authorized
                    //already, we run in a conrolled enviroment
                }
            }

            );
            gsiSocket.setWrapMode(GssSocket.SSL_MODE);
            // this is what causing the problem
            gsiSocket.startHandshake();

            deleg_cred = context.getDelegCred();
            gsiSocket.close();
        }
        catch (Throwable t)
        {
            esay(t);
            //we do not propogate this exception since some exceptions we catch
are not serializable!!!
            throw new Exception(t.toString());
        }

This is the code that is running on the server that delgates the credential (and
plays the role of the client in this situation):

  public static void  delegateCredential(java.net.InetAddress inetAddress,
    int port,GSSCredential credential,boolean fulldelegation)
    throws Exception {
        // say("createSocket("+inetAddress+","+port+")");
        Socket s =null;
        try {
            //   say("delegateCredentials() user credential is "+credential);
            GSSManager manager = ExtendedGSSManager.getInstance();
            org.globus.gsi.gssapi.auth.GSSAuthorization gssAuth = 
            org.globus.gsi.gssapi.auth.HostAuthorization.getInstance();
            GSSName targetName = gssAuth.getExpectedName(null,
inetAddress.getCanonicalHostName());
            ExtendedGSSContext context =
            (ExtendedGSSContext) manager.createContext(targetName,
                GSSConstants.MECH_OID,
                credential,
                GSSContext.DEFAULT_LIFETIME);
            context.setOption(GSSConstants.GSS_MODE,
            GSIConstants.MODE_GSI);
            context.requestCredDeleg(true);
            if(fulldelegation) {
                context.setOption(GSSConstants.DELEGATION_TYPE,
GSIConstants.DELEGATION_TYPE_FULL);
            }
            else {
                context.setOption(GSSConstants.DELEGATION_TYPE,
GSIConstants.DELEGATION_TYPE_LIMITED);
            }
            //context.setOption(
            s = new Socket(inetAddress,port);
            GSIGssSocket gsiSocket = new GSIGssSocket(s, context);
            gsiSocket.setUseClientMode(true);
            gsiSocket.setAuthorization(
            //org.globus.ogsa.impl.security.authorization.HostAuthorization.
            gssAuth
            );
            gsiSocket.setWrapMode(GssSocket.SSL_MODE);
            gsiSocket.startHandshake();
        }
        catch(Exception e) {
            if(s!=null) {
                try {
                    s.close();
                }
                catch(Exception e1) {
                }
            }
            throw e;
        }
    }

------- Comment #2 From 2004-12-10 00:27:04 -------
This is a different problem you originally submitted this bug for.

Does getServerContext() return a new GSSContext instance each time? Is the 
server running for a while?

In general, I haven't seen this problem before and I'm not really sure what 
could be causing it. Can you maybe come up with some test case to replicate 
this problem?
------- Comment #3 From 2004-12-10 09:27:22 -------
Both sending and receiving parts are expected to run for months (here I mean
the
instance of java vm) 
On the receiving end  we do reuse the GlobusCredential instance, we call the
its
verify() method, to make sure it is valid, each time we reuse it, but the
GSSCredential and GSSContext are recreated each time we receive the delegation.

Would the new instance of the GlobusCredential for each session help?
On the receiving end
getServerContext() is implemented like this:


    private GSSContext getServerContext() throws GSSException {
        return SslGsiSocketFactory.getServiceContext(
        "/etc/grid-security/hostcert.pem",
        "/etc/grid-security/hostkey.pem",
        "/etc/grid-security/certificates");
    }


    public static GSSContext getServiceContext(
    String x509ServiceCert,
    String x509ServiceKey,
    String x509TrastedCACerts) throws GSSException {
        GSSCredential cred = getServiceCredential(x509ServiceCert,
x509ServiceKey,
        GSSCredential.ACCEPT_ONLY);

        if(trusted_certs == null) {
            trusted_certs =
            TrustedCertificates.load(x509TrastedCACerts);
        }

        GSSManager manager = ExtendedGSSManager.getInstance();
        ExtendedGSSContext context =
        (ExtendedGSSContext) manager.createContext(cred);

        context.setOption(GSSConstants.GSS_MODE,
        GSIConstants.MODE_GSI);
        context.setOption(GSSConstants.TRUSTED_CERTIFICATES,
        trusted_certs);
        return context;
    }


   public static GSSCredential getServiceCredential(
    String x509ServiceCert,
    String x509ServiceKey,int usage) throws GSSException {

        try {
            if(service_cred != null) {
                service_cred.verify();
            }
        }
        catch(GlobusCredentialException gce) {
            service_cred = null;

        }


        if(service_cred == null) {
            try {
                service_cred =new GlobusCredential(
                x509ServiceCert,
                x509ServiceKey
                );
            }
            catch(GlobusCredentialException gce) {
                throw new GSSException(GSSException.NO_CRED ,
                0,
                "could not load host globus credentials "+gce.toString());
            }
        }

        GSSCredential cred = new GlobusGSSCredentialImpl(service_cred, usage);

        return cred;
    }
------- Comment #4 From 2007-09-12 16:29:47 -------
Reassigning to current cog developer to close/fix as appropriate.