I should also note that the container freeze occurs at another point when the
DRS checks on the RFT status by calling it's GetResourceProperty operation. I
assume that the cause is the same as with the CreateRFT operation.

What JVM are you using? Also, try with a smaller request and see if you can 
still see the same behavior and try doing kill -QUIT. Also, check if the db 
itself is not a problem.

java version "1.4.2_06"

I will attempt to run smaller tests -- but it's important to note that the 
problem appears mostly at larger request sizes -- I can run tests including 
1000 files in a transfer request with 100%~ success. I'll let you know what I 
find.

As for the database, I'm not clear on what that has to do with it, since RFT 
doesn't even recieve any message to act on. The db shouldn't be an issue here.

I recreated the freeze with a file / transfer request size of 5000. The "kill
-QUIT" signal still has no effect on the process.

Just to demonstrate that the freeze results in the same concluding log messages
I'll copy them here again:

2005-06-21 10:43:31,411 DEBUG impl.TransferRequestWork
[Thread-13,_createTransfer:197] Attempting to create RFT resource
2005-06-21 10:43:32,145 DEBUG container.ServiceRequestQueue
[Thread-7,enqueue:52] Putting request in queue
2005-06-21 10:43:32,146 DEBUG container.ServiceDispatcher [Thread-7,run:196]
waiting threads: 2
2005-06-21 10:43:32,147 DEBUG container.ServiceRequestQueue
[Thread-8,dequeue:32] Getting request from queue
2005-06-21 10:43:32,148 DEBUG container.GSIServiceThread [Thread-8,process:63]
Thread-8:
processing requests
2005-06-21 10:43:32,392 DEBUG container.GSIServiceThread [Thread-8,process:95]
Thread-8:
Authenticated globus user: /C=US/O=Globus Alliance/OU=User/CN=fc5bc4e4ca.7e1f4757
2005-06-21 10:43:32,393 DEBUG container.ServiceThread [Thread-8,process:323]
Thread Thread-8 processing requests
2005-06-21 10:43:32,489 DEBUG container.ServiceThread
[Thread-8,parseRequest:769] ServiceThread: requested file name =
'wsrf/services/ReliableFileTransferFactoryService'
 

Are you doing kill -QUIT on the right process? What's your log4j settings? 
Maybe you should enable all logging for all classes.

Yes, in fact I'm doing 'kill -QUIT' on a few pid's -- I first try the pid that 
shows up in top as taking 99.9 CPU, then I try it's parent, and then it's 
parent's parent (which is the root of the process tree as shows by pstree).

My logging is what I mention in my first report:
org.globus.wsrf.container=DEBUG

And I also have:
org.globus.replica=DEBUG
org.globus.transfer=DEBUG

The latter two just provide context. By "all logging for all classes", I 
assume that means:
org.globus=DEBUG

If you mean something else, please let me know.

Change 'rootCategory' to 'DEBUG'. That will generate a lot of output but might 
help. 

After changing the log4j setting per your instruction, and then recreating the 
freeze, the following log is produced (last lines):

2005-06-21 13:45:58,251 DEBUG message.MessageElement [Thread-11,<init>:242]   
soapenv:mustUnderstand = '0'
2005-06-21 13:45:58,252 DEBUG i18n.ProjectResourceBundle [Thread-
14,handleGetObject:72] org.apache.axis.i18n.resource::handleGetObject
(startElem00)
2005-06-21 13:45:58,254 DEBUG i18n.ProjectResourceBundle [Thread-
11,handleGetObject:72] org.apache.axis.i18n.resource::handleGetObject
(pushHandler00)
2005-06-21 13:45:58,255 DEBUG encoding.SerializationContext [Thread-
14,startElement:991] Start element 
[http://www.globus.org/namespaces/2004/10/rft]:sourceUrl
2005-06-21 13:45:58,257 DEBUG encoding.DeserializationContext [Thread-
11,pushElementHandler:796] Pushing handler 
org.apache.axis.message.SOAPHandler@334bfc
2005-06-21 13:45:58,258 DEBUG utils.NSStack [Thread-14,push:75] NSPush (32)
2005-06-21 13:45:58,259 DEBUG encoding.DeserializationContext [Thread-
11,pushNewElement:765] Pushing element To
2005-06-21 13:45:58,261 DEBUG i18n.ProjectResourceBundle [Thread-
14,handleGetObject:72] org.apache.axis.i18n.resource::handleGetObject
(endElem00)
2005-06-21 13:45:58,262 DEBUG utils.NSStack [Thread-11,push:75] NSPush (32)
2005-06-21 13:45:58,264 DEBUG encoding.SerializationContext [Thread-
14,endElement:1097] End element sourceUrl
2005-06-21 13:45:58,265 DEBUG encoding.DeserializationContext [Thread-
11,startElement:1067] Exit: DeserializationContext::startElement()
2005-06-21 13:45:58,266 DEBUG utils.NSStack [Thread-14,pop:109] NSPop (32)
2005-06-21 13:45:58,267 DEBUG encoding.DeserializationContext [Thread-
11,endElement:1078] Enter: DeserializationContext::endElement
(http://schemas.xmlsoap.org/ws/2004/03/addressing, To)
2005-06-21 13:45:58,269 DEBUG i18n.ProjectResourceBundle [Thread-
14,handleGetObject:72] org.apache.axis.i18n.resource::handleGetObject
(startElem00)
2005-06-21 13:45:58,270 DEBUG i18n.ProjectResourceBundle [Thread-
11,handleGetObject:72] org.apache.axis.i18n.resource::handleGetObject
(popHandler00)
2005-06-21 13:45:58,272 DEBUG encoding.SerializationContext [Thread-
14,startElement:991] Start element 
[http://www.globus.org/namespaces/2004/10/rft]:destinationUrl
2005-06-21 13:45:58,273 DEBUG encoding.DeserializationContext [Thread-
11,popElementHandler:829] Popping handler 
org.apache.axis.message.SOAPHandler@334bfc

Let me know if you'd like the entire log attached to this record.

Yes, please attach the entire log.

This URL points to the complete log (bugzilla doesn't seem to want to accept 
it as an attachment):

http://www.isi.edu/~schuler/container.log.ned-0.200506211400.tar.bz2

I don't see anything really wrong in the log. Two interesting things but 
unrelated to the problem: 1) it looks like you are using GSI Sec Conv 
authentication on top of https, 2) I'm not seeing any output from 
MessageLoggingHandler, maybe logging for it was just explicitely disabled.

A few other things to try:
1) Ensure you are working with latest code from cvs (if not already)
2) Try with jdk 1.4.2_08 and if that does not help with jdk 1.5

I started by testing with j2sdk1.4.2_08 and jdk1.5.0_03. With j2sdk1.4.2_08 
the results are the same. It still hangs and very consistently.

However, the good news is that with jdk1.5.0_03 I cannot recreate the hang. 
The cpu spikes but interestingly the cpu usage seems to be spread across two 
or three java processes, the sum of which would be near 100%. Also, at times, 
I see the cpu spike on a single java process up to 96%~ but doesn't top out at 
99.9 and freeze. I'm not sure what improvements Sun has made to the JVM since 
1.4, but it appears to mitigate this problem.

I'm going to continue to run some tests (with jdk1.5) to make sure I cannot 
recreate the cpu spike during the GetRP call.

Reassigning to current wsrf developer to close/fix as appropriate