Here are the things we improved so far in working with large CRAB runs at UNL:

1) Fixed a recovery bug.
2) Updated to the latest WS-GRAM globus_4_0_branch code. This improved error
reporting making it easier to diagnose problems.
3) Updated to the latest RFT globus_4_0_branch code. This fixed some problems
with out-of-order transfers and hung GridFTP control channel problems.
4) Improved GRAM job queue utilization.  This allows for faster response times
for simple fork jobs. It also improves performance in general because relatively
fast events aren't stuck behind relatively slow staging events.
5) Implemented local transport to RFT. This improved container responsiveness
since it essentially removed RFT callouts from the list of total connection
attempts at any particular time, leaving Condor-G with more available threads to
submit jobs.

Current status:

CRAB runs up to about 2000 jobs are completeing for the most part. Occasionally
a job is left unsubmitted or is held because of an error.

CRAB runs above 2000 jobs aren't fairing so well. There's an issue where jobs
seem to be lost in the state machine. I can't find any direct evidence (perhaps
partly because the log files get so big it's hard to find anything out of the
ordinary), but this may be a delegation problem. Close to the end of the run
(i.e. when no more activity is observed in the container), there are a lot of
errors being generated pertaining to delegated credential resources that can't
be found. This is most likely because Condor-G failed to set the lifetime of the
delegated credentials to a long enough period of time to accomodate the long
execution time of the CRAB run.

Once the Condor guys fix this then we can resume testing to see if that helped
stability. If it does then we need to figure out a better way to handle the
situation so that it's easier to identify and doesn't simply result in lost
jobs. If not, obviously we'll have to keep looking for the cause.

I fixed a bug that seemed to eliminate the problem of jobs seemingly
disappearing from the state machine. In fact, there was evidence of this in the
form of a debug message rather than an error or warning. This is unrelated
afterall to the expiring delegation issue.

Jaime Frey said that he is testing the latest Condor-G code to make sure it's
doing what it's supposed to in terms of refreshing the delegated credential.
Meanwhile, Brian is installing the latest Condor-G release available since
fixes that may have an affect on the delegated credential issue are in the
latest release. We should have some results by weeks end at the latest.

I'm adding bug #3121 as a dependency since a large number of jobs seem to be
failing due to problems connecting to GridFTP. This bug pertains to the broken
check on the maximum number of transfers that are active at any one time.
Hopefully limiting the number of transfers will reduce the traffic between the
two machines involved and prevent connection timeouts.

There were condor-g issues in 6.7.14 with refreshing a delegation for  
long CRAB runs (> 12 hours I think).  This was hoped to be fixed with  
a new version of condor-g 6.7.17, but Brian backed that out as it  
caused problems with pre-ws gram users on the same machine ([condor-admin
#13508]).  Jaime is helping Brian to setup 2 condor installs - 1 for pre-ws
jobs and one for experimental use like this ws gram testing.

At the same time, the CRAB application is no longer stable and cannot  
be used for testing at the moment and, according to Brian, maybe not for a  
while.  We are stepping back to a dummy stage-in-sleep-stage-out job  
(condor-g-ws-test-sleep-io).  Initial testing with a 4 MB input and  
10MB output file produced RFT/gridftp errors.  We will reduce the  
file sizes and see where the breaking point is and then report back.  During 
this run there were also condor job execution errors that may have been caused
by NFS.  Brian has changed an NFS setting, hopefully avoiding the problem.  
We'll see.

Here are the current action items:

1) Brian: Get Condor-G version 6.7.17 running just for ws-gram jobs  
on osg-test2.
2) Peter: Adjust the input file size to 0.5 MB for condor-g-ws-test- 
sleep-io test.
3) Peter: Adjust the output file size to 1 MB for condor-g-ws-test- 
sleep-io test.
4) Peter: Run a 3500-job test run with condor-g-ws-test-sleep-io.

2 & 3 are necessary because of the RFT bug (#3121) that prevents throttling
the number of active transfers. After getting the above test to  
work, we can worry about getting RFT patched and find a reasonable  
setting to avoid the GridFTP server timeouts.

Tasks 1 through 3 of the latest action list are done (thanks Brian!). I'm
running a 500-job test right now just to shake everything down. If that runs
cleanly I'll bump it up to 3500.

I haven't had a chance to track down problems I was having with larger runs.
Here's the the email I wrote regarding this problem from March 31st:

>> I was able to run a 500-job test, but the 3500-job test had  
>> problems. It
>> got hung up at one point waiting for the fileCleanUp transfer request
>> before being destroyed. I looked into it and if I'm not mistaken it
>> looks like RFT lost the request. Here's the line where GRAM registers
>> the transfer request with the RFT notification listener thread:
>>
>> 2006-03-31 12:41:49,653 DEBUG exec.StagingListener [RunQueue
>> FileCleanUp,registerTransferJob:104]
>> [execJobKey:72193bb0-c0b9-11da-9560-fa7a7cd61e10,transferJobKey: 
>> 150037]
>> Leaving registerTransferJob()
>>
>> From this I determined that the request ID is  
>> "150037" ("transferJobKey"
>> is my name for the RFT request id--don't worry, I checked for  
>> sanity's
>> sake that this wasn't a transfer ID too). I then did a search in the
>> MySQL database for this request:
>>
>> mysql> select * from transfer where request_id=150037;
>> Empty set (0.12 sec)
>>
>> mysql> select * from request where id=150037;
>> Empty set (0.00 sec)
>>
>> RFT doesn't appear to delete the database records, so even if the
>> request was destroyed the record should still be there. That said, I
>> also don't see an deliver() calls for that request. Unfortunately I
>> didn't have debugging turned on for RFT, so I'll have to restart the
>> test.
>>
>> Peter

Hello

I have talked to Stu Martin who said follow up to this problem since April 28th
- see the bottom of this Bugzilla report - has been superceded by a higher
priority project. Stu also said that he hopes to get some effort available in
the next month or so.

Is this a correct assessment? When I read the last posting on this Bugzilla
thread I thought there would be continuing follow up and was worried that OSG
had failed to be avialable as needed.

Thank you

Ruth

I am marking this campaign as closed.  Another campaign bug 4506 has been
created to focus the continuing effort in this area.

-Stu

I'm increasing the priority of the replacement campaign aimed at this issue,
4050.

Oops! I mean the priority of campaign 4506 is being increased!