If a client adds a job credential endpoint to the job descripton the
credential is fetched as part of the job creation Web Services call 
(MJFS.createManagedJob()).

Things that happen there:
 1. Check if the credential has already been fetched from the
    DelegationService. If so use the cached credential. If not fetch it
    from the DelegationService and cache it.
 2. Create a user proxy file and store it.
    This is a quite expensive operation since a call to a shellscript
    (globus-gram-local-proxy-tool) is done which creates the proxy-file.
    Also 2 threads are involved in this.

If a lot of job creation requests come in at a time the by far the biggest 
amount of time spent in job creation is spent while fetching the credential
and creating the user proxy (about 95%). Under moderate submission load on a
strong machine this can take about 15 seconds.

This probably explains the submit timeouts OSG and LEAD folks found in their
simple tests on weaker hosts.

Condor-G by default delegates a job credential.

It seems unnecessary to fetch the credential and create the user-proxy as
part of the job resource creation. This can be done in a normal job processing
step in the StateMachine.
Tests showed that this reduces the time to create a job to 20-1000 ms.
This also ensures that only a limited amount of user-proxy creations happen
at a time. The max number is defined by the number of processing threads in
gram configured in JNDI.

Tests showed that the time to create a job resource was quite independent
from the job creation request load when the job credential was not fetched
during resource creation.

In 4.2 the job credential will be fetched as first processing state of a job
resource (internal state None)