[HADOOP-7888] TestFailoverProxy fails intermittently on trunk

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.24.0
• Fix Version/s: 2.0.0-alpha
• Component/s: test
• Target Version/s: 0.23.3, 0.24.0
• Hadoop Flags: Reviewed
• Assignee: Jason Darrell Lowe
• Reporter: Jason Darrell Lowe
• Created: 07/Dec/11 01:54
• Updated: 23/May/12 20:15
• Resolved: 08/Dec/11 01:21

Description

TestFailoverProxy can fail intermittently with the failures occurring in testConcurrentMethodFailures(). The test has a race condition where the two threads may be sequentially invoking the unreliable interface rather than concurrently. Currently the proxy provider's getProxy() method contains the thread synchronization to enforce a concurrent invocation, but examining the source to RetryInvocationHandler.invoke() shows that the call to getProxy() during failover is too late to enforce a truly concurrent invocation.

For this particular test, one thread could race ahead and block on the CountDownLatch in getProxy() before the other thread even enters RetryInvocationHandler.invoke(). If that happens the second thread will cache the newly updated value for proxyProviderFailoverCount, since the failover has mostly been processed by the original thread. Therefore the second thread ends up assuming no other thread is present, performs a failover, and the test fails because two failovers occurred instead of one.