Note 0: I'll attach a diagram that gives an overview over the scenario and
        may help to understand what i describe below.
Note 1: "Notification" is not meant in the WSRF context here.
Note 2: The following may not be totally exact, but shall illustrate
        what happens. Also I'm talking about a Condor pool instead
        of a more abstract "LocalResourceManager".
Note 3: GREEN, BLUE, RED refer to the colors in the attached document

What happens:
#############
Under the current circumstances it happens sometimes that notifications
about job states from the SchedulerEventGenerator to the JobStateMonitor
get lost. This causes some jobs to keep stuck in the container.

A RunThread submits a job to the condor pool which writes the status of the
job to its logfile. The SchedulerEventGenerator monitors that logfile
and tells the JobStateMonitor about changes in the jobs state in the Condor
pool (BLUE).

After a job is sent to the Condor pool, the job resource is registered
at the JobStateMonitor via the JobStateMonitorSubscriptionManager. The
registration at the JobStateMonitorSubscriptionManager is a non-blocking
action. The JobStateMonitorSubscriptionManager takes registration requests
by all jobs, queues them if necessary and registers them at the
JobStateMonitor on behalf of the job resources (GREEN).

It sometimes happens that events of a job are sent to the JobStateMonitor
from the SchedulerEventGenerator (BLUE) *before* the job resource had been
registered at the JobStateMonitor (GREEN). In that case, the events can't be
sent to the MEJobHome direcJobStateMonitorSubscriptionManagertly but are saved
in the EventCache. When the registration comes in, the cached events are
triggered to the MEJobHome.

Additionally a flushCache Task periodically removes all cached events that
are older than N milliseconds (RED).

If a registration for job state changes (GREEN) takes place *after* all
events of a job are cached (BLUE) and if the events are cached for longer
than N milliseconds, the job resource will never be notified about job state
changes, since all events had been removed by the flushCache task (RED).


Reason why this happens:
########################

* Short running jobs and good throughput from GRAM4 to the Condor-pool
  => many job state change events enter the JobStateMonitor at the same time.

* The JobStateMonitorSubscriptionManager queues incoming registration
  requests (RED). The registration at the JobStateMonitor takes from
  about 0 to 20 seconds.
  This causes heaping up registration requests in the queue and explains
  why the registration is delayed: requests are heaped up and stay there for
  up to 35 minutes before they get registered at the JobStateMonitor.


Some notes and questions:
#########################


1. why is job submission (BLUE) so fast and registration so slow.
   The BLUE way should take much more time than the GREEN way.
   Is it that the container is so busy that Threads don't get
   much execution time (see item 2).
   I tried to improve performance of the JobStateMonitor and things became
   slightly better, but this just shifted the problem backwards and did
   not eliminate it.
   (What i did was a change of some synchronization issues and removal
   of the DispatcherThread in JobStateMonitor which starts a Thread for each
   notification)

2. I found that the registration is slow only until all jobs of a large
   job submission are submitted by Condor-G i.e. all jobs are either done
   or in the container.
   If all jobs are in the container then things speed up noticeably.
   I saw this correlation earlier: During the submission phase all
   tasks in the container are slower.
   What could cause this?
   * Do Container-Threads have higher priority?
   * Is the system busy with encryption and decryption of messages?
   ... 


Possible solutions:
###################

* go the GREEN way before the BLUE way and use blocking subscription
  instead of non-blocking.

* avoid the flushCache task. I don't know exactely if this can be done
  although the logfile which is used by the Condor pool contains only entries
  for jobs submitted by GRAM. But AFAIK this is not the case with other
  local resource managers like LSF, PBS, ...