Bugzilla – Bug 4927
Last modified: 2007-06-05 09:57:04
You need to
before you can comment on or make changes to this bug.
RFT can become more reliable by improving the fault tolerance of the tool.
Right now, each instance of RFT has its own database and is running in its own
container. If a particular container or database goes down, then any transfers
that RFT instance was processing will be stopped. A more fault tolerant model
would allow for another RFT instance to pick up the requests that had been
running on the downed instance and continue to process them.
Created an attachment (id=1147) [details]
Changes to RFT code to support enhancement
Created an attachment (id=1148) [details]
Changes to Persistent Subscription code to support enhancement
Created an attachment (id=1149) [details]
Changes to Delegation Resource code
Created an attachment (id=1150) [details]
Changes to Transfer Work code
Created an attachment (id=1151) [details]
Changes to Transfer Client code
Created an attachment (id=1152) [details]
New code for RFT
Created an attachment (id=1153) [details]
New code for fault tolerant RFT
Created an attachment (id=1154) [details]
Changes to Resource Home Impl code
Created an attachment (id=1155) [details]
Changes to Reliable File Transfer Impl code
Created an attachment (id=1156) [details]
Changes to Reliable File Transfer DB code
Created an attachment (id=1157) [details]
Changes in the database that are needed to support code
The files submitted contain the code developed at NCSA to support the fault
tolerance talked about by this enhancement. The assumption made is that all of
the RFT instances will be sharing a MySQL database. The initial code uses a
single database for all instances but in the future it is hoped that the code
can be moved to a more fault tolerant DB solution, possibly either a clustered
DB or a replicated DB.
When the new RFT code starts up with a container a timer is started. This
timer is used to check the status of running transfers. Right now the code is
set to check every 30 seconds, this can be changed and probably should be moved
to a jndi file for user configuration. A side benefit to this timer is the
ability to recover from a down database. If the database is taken off line for
some reason but the container continues running, each time the timer fires off
it checks the DB connection. When the DB comes back on line, the timer will
reestablish the connection.
Under normal operations the timer will perform a query on the database to see
if any transfers are stalled. This is accomplished by adding new field to the
restart table to hold the last time the row was touched. An assumption is made
at this point that if X amount of time has passed and the request is still
active then the transfer is stalled. At this point another container can pick
up this request and begin to process it. A container ID field has also been
added to the request table. This is to prevent containers from picking up each
others requests. The time is hard coded at this point and should be moved to a
jndi file for user configuration.
In order to fully facilitate this fail over, changes needed to be made to
Delegation Services and the handling of notifications. If another container
picks up a stalled transfer it needs to know about the DS information. In
current Globus code, the DS is persisted to a flat file. To allow other
containers access to this information, DS has been changed to persist its self
to a DB table.
The same is true for the notification code. In order for notifications to get
back to the correct location a new container needs to now the information. As
with DS, notifications are currently being persisted to a flat file. Under the
new code the notifications are now being persisted to DB tables.
The code that has been submitted to handle the DS and notification changes
needs some changes. The current code has hard coded connections to the
database that was used for development. This code needs to be removed and
should be made configurable like the rest of the Globus DB code.
1) Would it be possible for you to post unified diffs, rather than drop-in
2) The DelegationService code you present appears to be using file persistence
- where's the database code?
3) There would seem to be problems for long-running transfers that require
proxy refreshes - the client would not know that the delegation resource in use
for the transfer had moved. That is, the problem of "fault tolerant Delegation
Service" seems to still exist in this scenario unless one has done some kind of
DNS/load-balanced IP readdressing the the second container?
I'm just commenting as an interested observer - Ravi is already on vacation and
should be back in January.
(In reply to comment #13)
> 3) There would seem to be problems for long-running transfers that require
> proxy refreshes - the client would not know that the delegation resource in use
> for the transfer had moved. That is, the problem of "fault tolerant Delegation
> Service" seems to still exist in this scenario unless one has done some kind of
> DNS/load-balanced IP readdressing the the second container?
Right, this code would work best in combination with a front-end load-balancer
and a back-end highly-available (clustered) database. Since the DS resources
are shared across the instances via the database, the client could contact any
of the containers to refresh the credential.
However, my understanding is that Patrick doesn't yet have the DS refresh
callback working when the request fails over to another container. (Patrick,
correct me if I'm wrong.)
Patrick is working on improvements, so please keep the comments/suggestions
(In reply to comment #14)
> Right, this code would work best in combination with a front-end load-balancer
Alternatively, WS-Naming could be used to locate the moved resources.
Created an attachment (id=1174) [details]
Changes made to the DS code to support new RFT code.
Created an attachment (id=1175) [details]
Changes made to RFT code to support new failover functionality.
Created an attachment (id=1176) [details]
WSRF subscription changes
Changes made to the PersistentSubscription code in support of new RFT code.
Added files that contain cvs diff -u information.
Again, it should be noted that the PersistentSubscription and
ReliableFileTransferResource have hard coded calls to an NCSA test database.
This will be corrected and moved to a JNDI file.
We're presenting a paper on this work at TeraGrid '07
(http://www.union.wisc.edu/teragrid07/). The paper is available online at:
Patrick has submitted separate bugs for modifications to the different
Bug 5255 - RFT
Bug 5256 - Delegation Service
Bug 5257 - GT Core Persistence API
The latest patches can be found there.