Bug 2749 – concurrent modification error on concurrent start() calls

Bug 2749 - concurrent modification error on concurrent start() calls

Summary:

concurrent modification error on concurrent start() calls

Status:	RESOLVED DUPLICATE of bug 2574

Product:	RFT
Component:	RFT
Version:	development
Platform:	All All

Importance:	P3 critical
Target Milestone:	---
Assigned To:	Ravi Madduri

URL:
Keywords:

Depends on:
Blocks:
	Show dependency tree / graph

Reported:	2005-02-16 10:37 by Peter Lane
Modified:	2005-02-16 11:45 (History)

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note

You need to log in before you can comment on or make changes to this bug.

Description From Peter Lane 2005-02-16 10:37:34

When I try to run throughput tests with more than one concurrent client
submitting staging jobs, I get the following error:

 faultString: java.rmi.RemoteException: Unable to create RFT resource; nested
exception is: 
????org.globus.transfer.reliable.service.exception.RftException: Error
processing delegated credential [Caused by: Failed to store resource; nested
exception is: 
????java.util.ConcurrentModificationException] [Caused by:  [Caused by: Failed
to store resource; nested exception is: 
????java.util.ConcurrentModificationException]]
 faultActor: 
 faultNode: 
 faultDetail: 
????{http://xml.apache.org/axis/}stackTrace:java.rmi.RemoteException: Unable to
create RFT resource; nested exception is: 
????org.globus.transfer.reliable.service.exception.RftException: Error
processing delegated credential [Caused by: Failed to store resource; nested
exception is: 
????java.util.ConcurrentModificationException] [Caused by:  [Caused by: Failed
to store resource; nested exception is:
????java.util.ConcurrentModificationException]]
????at
org.globus.transfer.reliable.service.factory.ReliableFileTransferFactoryService.createReliableFileTransfer(ReliableFileTransferFactoryService.java:218)

------- Comment #1 From Peter Lane 2005-02-16 11:28:13 -------

This is happening without concurrent clients also as long as the client is
managing multple jobs at the same time.

------- Comment #2 From Peter Lane 2005-02-16 11:35:50 -------

Here's the entire error message:

2005-02-16 11:21:32,370 ERROR throughput.ClientThread
[Thread-36,stateChanged:260] a job failed with handle
https://140.221.36.11:8443/wsrf/services/ManagedExecutableJobService?31d7edf0-803f-11d9-99b2-f4fa8b5db1cd:
fault type: org.globus.exec.generated.StagingFaultType:
attribute: fileStageOut
command: StageOut
description:
Staging error for RSL element fileStageOut.
faultReason: 
faultString: 
gt2ErrorCode: 0
originator: Address:
https://140.221.36.11:8443/wsrf/services/ManagedJobFactoryService
Reference property[0]:
<ns1:ResourceID
xmlns:ns1="http://www.globus.org/namespaces/2004/10/gram/job">31d7edf0-803f-11d9-99b2-f4fa8b5db1cd</ns1:ResourceID>

stackTrace:
org.globus.exec.generated.StagingFaultType: Staging error for RSL element
fileStageOut.
Timestamp: Wed Feb 16 11:21:31 CST 2005
Originator: Address:
https://140.221.36.11:8443/wsrf/services/ManagedJobFactoryService
Reference property[0]:
<ns1:ResourceID
xmlns:ns1="http://www.globus.org/namespaces/2004/10/gram/job">31d7edf0-803f-11d9-99b2-f4fa8b5db1cd</ns1:ResourceID>

Caused by: org.oasis.wsrf.faults.BaseFaultType: AxisFault
 faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server.userException
 faultSubcode: 
 faultString: java.rmi.RemoteException: Unable to create RFT resource; nested
exception is: 
        org.globus.transfer.reliable.service.exception.RftException: Error
processing delegated credential [Caused by: Failed to store resource; nested
exception is: 
        java.util.ConcurrentModificationException] [Caused by:  [Caused by:
Failed to store resource; nested exception is: 
        java.util.ConcurrentModificationException]]
 faultActor: 
 faultNode: 
 faultDetail: 
        {http://xml.apache.org/axis/}stackTrace:java.rmi.RemoteException: Unable
to create RFT resource; nested exception is: 
        org.globus.transfer.reliable.service.exception.RftException: Error
processing delegated credential [Caused by: Failed to store resource; nested
exception is: 
        java.util.ConcurrentModificationException] [Caused by:  [Caused by:
Failed to store resource; nested exception is: 
        java.util.ConcurrentModificationException]]
        at
org.globus.transfer.reliable.service.factory.ReliableFileTransferFactoryService.createReliableFileTransfer(ReliableFileTransferFactoryService.java:218)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:324)
        at
org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:384)
        at
org.globus.axis.providers.RPCProvider.invokeMethodSub(RPCProvider.java:104)
        at
org.globus.axis.providers.PrivilegedInvokeMethodAction.run(PrivilegedInvokeMethodAction.java:39)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:379)
        at org.globus.gsi.jaas.GlobusSubject.runAs(GlobusSubject.java:49)
        at org.globus.gsi.jaas.JaasSubject.doAs(JaasSubject.java:84)
        at org.globus.axis.providers.RPCProvider.invokeMethod(RPCProvider.java:94)
        at
org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:281)
        at org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:319)
        at
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
        at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
        at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
        at org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:450)
        at org.apache.axis.server.AxisServer.invoke(AxisServer.java:285)
        at org.globus.wsrf.container.ServiceThread.doPost(ServiceThread.java:647)
        at org.globus.wsrf.container.ServiceThread.process(ServiceThread.java:378)
        at
org.globus.wsrf.container.GSIServiceThread.process(GSIServiceThread.java:124)
        at org.globus.wsrf.container.ServiceThread.run(ServiceThread.java:281)
Caused by: org.globus.transfer.reliable.service.exception.RftException: Error
processing delegated credential [Caused by: Failed to store resource; nested
exception is: 
        java.util.ConcurrentModificationException] [Caused by:  [Caused by:
Failed to store resource; nested exception is: 
        java.util.ConcurrentModificationException]]
        at
org.globus.transfer.reliable.service.ReliableFileTransferResource.processDelegatedCredential(ReliableFileTransferResource.java:384)
        at
org.globus.transfer.reliable.service.ReliableFileTransferResource.&lt;init&gt;(ReliableFileTransferResource.java:232)
        at
org.globus.transfer.reliable.service.ReliableFileTransferHome.create(ReliableFileTransferHome.java:108)
        at
org.globus.transfer.reliable.service.factory.ReliableFileTransferFactoryService.createReliableFileTransfer(ReliableFileTransferFactoryService.java:213)
        ... 23 more

        {http://xml.apache.org/axis/}hostname:ruly.mcs.anl.gov

java.rmi.RemoteException: Unable to create RFT resource; nested exception is: 
        org.globus.transfer.reliable.service.exception.RftException: Error
processing delegated credential [Caused by: Failed to store resource; nested
exception is: 
        java.util.ConcurrentModificationException] [Caused by:  [Caused by:
Failed to store resource; nested exception is: 
        java.util.ConcurrentModificationException]]
        at
org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:221)
        at
org.apache.axis.message.SOAPFaultBuilder.endElement(SOAPFaultBuilder.java:128)
        at
org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1087)
        at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
        at
org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanEndElement(Unknown Source)
        at
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown
Source)
        at
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
        at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
        at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
        at javax.xml.parsers.SAXParser.parse(SAXParser.java:345)
        at
org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)
        at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:645)
        at org.apache.axis.Message.getSOAPEnvelope(Message.java:424)
        at
org.apache.axis.message.addressing.handler.AddressingHandler.processClientResponse(AddressingHandler.java:305)
        at
org.apache.axis.message.addressing.handler.AddressingHandler.invoke(AddressingHandler.java:110)
        at
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
        at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
        at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
        at org.apache.axis.client.AxisClient.invoke(AxisClient.java:190)
        at org.apache.axis.client.Call.invokeEngine(Call.java:2726)
        at org.apache.axis.client.Call.invoke(Call.java:2709)
        at org.apache.axis.client.Call.invoke(Call.java:2385)
        at org.apache.axis.client.Call.invoke(Call.java:2308)
        at org.apache.axis.client.Call.invoke(Call.java:1765)
        at
org.globus.rft.generated.bindings.ReliableFileTransferFactoryPortTypeSOAPBindingStub.createReliableFileTransfer(ReliableFileTransferFactoryPortTypeSOAPBindingStub.java:874)
        at
org.globus.exec.service.exec.StateMachine.submitStagingRequest(StateMachine.java:2303)
        at
org.globus.exec.service.exec.StateMachine.processStageOutState(StateMachine.java:1546)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:324)
        at
org.globus.exec.service.exec.StateMachine.processState(StateMachine.java:323)
        at org.globus.exec.service.exec.RunQueue.run(RunQueue.java:141)
Timestamp: Wed Feb 16 11:21:31 CST 2005

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:274)
        at java.lang.Class.newInstance0(Class.java:308)
        at java.lang.Class.newInstance(Class.java:261)
        at
org.apache.axis.encoding.ser.BeanDeserializer.<init>(BeanDeserializer.java:90)
        at
org.apache.axis.encoding.ser.BeanDeserializer.<init>(BeanDeserializer.java:76)
        at
org.globus.exec.generated.StagingFaultType.getDeserializer(StagingFaultType.java:152)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:324)
        at
org.apache.axis.encoding.DeserializationContext.getDeserializerForClass(DeserializationContext.java:510)
        at
org.apache.axis.encoding.ser.BeanDeserializer.onStartChild(BeanDeserializer.java:250)
        at
org.apache.axis.encoding.DeserializationContext.startElement(DeserializationContext.java:1035)
        at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source)
        at
org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
        at
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown
Source)
        at
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
        at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
        at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
        at javax.xml.parsers.SAXParser.parse(SAXParser.java:345)
        at
org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)
        at
org.globus.wsrf.encoding.ObjectDeserializer.toObject(ObjectDeserializer.java:56)
        at org.globus.exec.client.GramJob.deliver(GramJob.java:1444)
        at
org.globus.wsrf.impl.notification.NotificationConsumerProvider.notify(NotificationConsumerProvider.java:106)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:324)
        at
org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:384)
        at
org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:281)
        at org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:319)
        at
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
        at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
        at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
        at org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:450)
        at org.apache.axis.server.AxisServer.invoke(AxisServer.java:285)
        at org.globus.wsrf.container.ServiceThread.doPost(ServiceThread.java:647)
        at org.globus.wsrf.container.ServiceThread.process(ServiceThread.java:378)
        at
org.globus.wsrf.container.GSIServiceThread.process(GSIServiceThread.java:124)
        at org.globus.wsrf.container.ServiceThread.run(ServiceThread.java:281)

stateWhenFailureOccurred: StageOut
timestamp:
java.util.GregorianCalendar[time=1108574491640,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2005,MONTH=1,WEEK_OF_YEAR=8,WEEK_OF_MONTH=3,DAY_OF_MONTH=16,DAY_OF_YEAR=47,DAY_OF_WEEK=4,DAY_OF_WEEK_IN_MONTH=3,AM_PM=1,HOUR=5,HOUR_OF_DAY=17,MINUTE=21,SECOND=31,MILLISECOND=640,ZONE_OFFSET=0,DST_OFFSET=0]
Message:
null

------- Comment #3 From Peter Lane 2005-02-16 11:42:38 -------

Looks like this is in the Delegation Service.  Marking as a duplicate of #2574.

*** This bug has been marked as a duplicate of 2574 ***

------- Comment #4 From Peter Lane 2005-02-16 11:43:35 -------

Woops...marking as a duplicate of #2575.

*** This bug has been marked as a duplicate of 2575 ***

*** This bug has been marked as a duplicate of 2575 ***

------- Comment #5 From Ravi Madduri 2005-02-16 11:45:47 -------

Looks like the error is coming from DelegationUtil.registerListener() in
processDelegatedCredentials() in 
RFT resource impl. GRAM seems to be reusing delegation listeners to solve a
similar problem.