We handle the deletion of non-existing user proxy files very badly,
in several ways.
If a job tries to fetch a non-existing credential (probably destroyed
earlier) from the delegation service and then, because the credential
does not exist anymore, tries to delete the user proxy file created
from that credential earlier, which does not exist either, because it
was probably deleted when the credential was destroyed, then we end
up in an infinite loop and a blocking situation that can block all
job requests of a particular user.
A not completely uncommon situation i guess.

The key problem is this code:

Process proc = Runtime.getRuntime().exec(command);
proc.getOutputStream().close();

// handle possible sudo password prompt
BufferedReader stderr =
    new BufferedReader(
        new InputStreamReader(proc.getErrorStream()));
String stderrLine = stderr.readLine();
while (stderrLine != null) {
    if (stderrLine.trim().indexOf("Password:") == 0) { 
        String errorMessage =
            i18n.getMessage(Resources.MISCONFIGURED_SUDO,
                new String[] { "globus-gram-local-proxy-tool",
                    this.localUserId });
        throw new RuntimeException(errorMessage);
    }
}

We are trying to delete the user proxy file in this command. If the
file does not exist anymore, then the stderr output does not contain
"Password:", but something like "rm ... file does not exist". In this
situation we never leave the loop.

Another problem is that this code is executed while we are holding a
lock on userInstances, which causes that the lock is never released.
Any credential check for this particular user will hang then too, because
userInstances is involved.

The infinite loop should be avoided. A quick fix is to modify
$GLOBUS_LOCATION/libexec/globus-gram-local-proxy-tool
to not fail if the user proxy file does not exist: Replace

    -delete)
        # proxyfile should exist
        exec rm "$PROXYFILE"
        exit $?
        ;;

by

    -delete)
        if [ -e "$PROXYFILE" ]; then
            exec rm "$PROXYFILE"
            exit $?
        else
            exit 0
        fi
        ;;

In general, the whole freaky code in DelegatedCredential and behavior
in ws-gram in situations like the above should be reviewed.