I figured out why the "Authentication failed" exception was happening. The main
reason was that I was creating the GSSContext with this command: 
ExtendedGSSContext context =
            (ExtendedGSSContext) manager.createContext(credential);

When in fact I was supposed to use this one:            

(ExtendedGSSContext) manager.createContext(peer,
                GSSConstants.MECH_OID,
                credential,
                GSSContext.DEFAULT_LIFETIME);

It has nothing to do with the GSI Socket Authentication, it is an internal peer
name authentication step, that was not present in the earlier releases, that was
causing this problem. 

So now we observe a different behaviour, the delegation step works with most of
the peers most of the time, but then suddenly it starts to fail with a
mestirious "Decrypt Error" failure on the receiving side. This behaviour
continues for some time, then it goes away. We obesrved it with the previos
version of jglobus libraries (built out of cvs last spring), and we were hoping
to see this behavior gone with the updated jglobus. But we still see this. We
really hope you will help us, because it is a real show stopper for us. Here are
the details. 
1. The credential is delegated from the client to the server for performance of
the gridftp transfers by the server on behalf of the client. We never observed
any problems on this step of delegation, we perform full delegation in this step. 
2. The delegated credential is delegated from the server to one of the multiple
disk nodes, where it is to be utilized for performing the gridftp transfer. Once
the credntial is delegated to the disk node, we never see any problems with
gridftp transfer step. But unfortunately the step of second delegation is the
one that fails sometimes.
Here is the exception we see on the disk node, receiving the credential:

org.globus.common.ChainedIOException: Authentication failed [Caused by: Failure
unspecified at GSS-API level [Root error message: Decrypt error]]. Caused by
GSSException: Failure unspecified at GSS-API level [Root error message: Decrypt
error].  Root exception is COM.claymoresystems.ptls.SSLThrewAlertException:
Decrypt error
     at COM.claymoresystems.ptls.SSLConn.alert(SSLConn.java:235)
     at
COM.claymoresystems.ptls.SSLCertificateVerify.decode(SSLCertificateVerify.java:160)
     at
COM.claymoresystems.ptls.SSLHandshakeServer.recvCertificateVerify(SSLHandshakeServer.java:561)
     at
COM.claymoresystems.ptls.SSLHandshakeServer.processTokens(SSLHandshakeServer.java:228)
     at
COM.claymoresystems.ptls.SSLHandshake.processHandshake(SSLHandshake.java:135)
     at
org.globus.gsi.gssapi.GlobusGSSContextImpl.acceptSecContext(GlobusGSSContextImpl.java:276)
     at org.globus.gsi.gssapi.net.GssSocket.authenticateServer(GssSocket.java:119)
     at org.globus.gsi.gssapi.net.GssSocket.startHandshake(GssSocket.java:137)
     at
diskCacheV111.movers.RemoteGsiftpTransferProtocol_1.runIO(RemoteGsiftpTransferProtocol_1.java:265)
     at
diskCacheV111.pools.MultiProtocolPool2$RepositoryIoHandler.run(MultiProtocolPool2.java:1154)
     at diskCacheV111.util.SimpleJobScheduler$SJob.run(SimpleJobScheduler.java:63)
     at java.lang.Thread.run(Thread.java:534)


Here is the code that is used to receive the credential:

        Socket deleg_socket = ss.accept();
        GSSCredential deleg_cred ;
        try
        {
            GSSContext context = getServerContext();
            GSIGssSocket gsiSocket = new GSIGssSocket(deleg_socket, context);
            gsiSocket.setUseClientMode(false);
            gsiSocket.setAuthorization(
            new Authorization() {
                public void authorize(GSSContext context, String host)
                throws AuthorizationException {
                    //we might add some authorization here later
                    //but in general we trust that the connection
                    //came from a head node and user was authorized
                    //already, we run in a conrolled enviroment
                }
            }

            );
            gsiSocket.setWrapMode(GssSocket.SSL_MODE);
            // this is what causing the problem
            gsiSocket.startHandshake();

            deleg_cred = context.getDelegCred();
            gsiSocket.close();
        }
        catch (Throwable t)
        {
            esay(t);
            //we do not propogate this exception since some exceptions we catch
are not serializable!!!
            throw new Exception(t.toString());
        }

This is the code that is running on the server that delgates the credential (and
plays the role of the client in this situation):

  public static void  delegateCredential(java.net.InetAddress inetAddress,
    int port,GSSCredential credential,boolean fulldelegation)
    throws Exception {
        // say("createSocket("+inetAddress+","+port+")");
        Socket s =null;
        try {
            //   say("delegateCredentials() user credential is "+credential);
            GSSManager manager = ExtendedGSSManager.getInstance();
            org.globus.gsi.gssapi.auth.GSSAuthorization gssAuth = 
            org.globus.gsi.gssapi.auth.HostAuthorization.getInstance();
            GSSName targetName = gssAuth.getExpectedName(null,
inetAddress.getCanonicalHostName());
            ExtendedGSSContext context =
            (ExtendedGSSContext) manager.createContext(targetName,
                GSSConstants.MECH_OID,
                credential,
                GSSContext.DEFAULT_LIFETIME);
            context.setOption(GSSConstants.GSS_MODE,
            GSIConstants.MODE_GSI);
            context.requestCredDeleg(true);
            if(fulldelegation) {
                context.setOption(GSSConstants.DELEGATION_TYPE,
GSIConstants.DELEGATION_TYPE_FULL);
            }
            else {
                context.setOption(GSSConstants.DELEGATION_TYPE,
GSIConstants.DELEGATION_TYPE_LIMITED);
            }
            //context.setOption(
            s = new Socket(inetAddress,port);
            GSIGssSocket gsiSocket = new GSIGssSocket(s, context);
            gsiSocket.setUseClientMode(true);
            gsiSocket.setAuthorization(
            //org.globus.ogsa.impl.security.authorization.HostAuthorization.
            gssAuth
            );
            gsiSocket.setWrapMode(GssSocket.SSL_MODE);
            gsiSocket.startHandshake();
        }
        catch(Exception e) {
            if(s!=null) {
                try {
                    s.close();
                }
                catch(Exception e1) {
                }
            }
            throw e;
        }
    }

This is a different problem you originally submitted this bug for.

Does getServerContext() return a new GSSContext instance each time? Is the 
server running for a while?

In general, I haven't seen this problem before and I'm not really sure what 
could be causing it. Can you maybe come up with some test case to replicate 
this problem?

Both sending and receiving parts are expected to run for months (here I mean
the
instance of java vm) 
On the receiving end  we do reuse the GlobusCredential instance, we call the
its
verify() method, to make sure it is valid, each time we reuse it, but the
GSSCredential and GSSContext are recreated each time we receive the delegation.

Would the new instance of the GlobusCredential for each session help?
On the receiving end
getServerContext() is implemented like this:


    private GSSContext getServerContext() throws GSSException {
        return SslGsiSocketFactory.getServiceContext(
        "/etc/grid-security/hostcert.pem",
        "/etc/grid-security/hostkey.pem",
        "/etc/grid-security/certificates");
    }


    public static GSSContext getServiceContext(
    String x509ServiceCert,
    String x509ServiceKey,
    String x509TrastedCACerts) throws GSSException {
        GSSCredential cred = getServiceCredential(x509ServiceCert,
x509ServiceKey,
        GSSCredential.ACCEPT_ONLY);

        if(trusted_certs == null) {
            trusted_certs =
            TrustedCertificates.load(x509TrastedCACerts);
        }

        GSSManager manager = ExtendedGSSManager.getInstance();
        ExtendedGSSContext context =
        (ExtendedGSSContext) manager.createContext(cred);

        context.setOption(GSSConstants.GSS_MODE,
        GSIConstants.MODE_GSI);
        context.setOption(GSSConstants.TRUSTED_CERTIFICATES,
        trusted_certs);
        return context;
    }


   public static GSSCredential getServiceCredential(
    String x509ServiceCert,
    String x509ServiceKey,int usage) throws GSSException {

        try {
            if(service_cred != null) {
                service_cred.verify();
            }
        }
        catch(GlobusCredentialException gce) {
            service_cred = null;

        }


        if(service_cred == null) {
            try {
                service_cred =new GlobusCredential(
                x509ServiceCert,
                x509ServiceKey
                );
            }
            catch(GlobusCredentialException gce) {
                throw new GSSException(GSSException.NO_CRED ,
                0,
                "could not load host globus credentials "+gce.toString());
            }
        }

        GSSCredential cred = new GlobusGSSCredentialImpl(service_cred, usage);

        return cred;
    }

Reassigning to current cog developer to close/fix as appropriate.