[HBASE-9085] Integration Tests fails because of bug in teardown phase where the cluster state is not being restored properly.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.95.0, 0.94.9, 0.94.10
• Fix Version/s: 0.98.0, 0.95.2, 0.94.11
• Component/s: test
• Hadoop Flags: Reviewed
• Assignee: gautam
• Reporter: gautam
• Created: 30/Jul/13 09:49
• Updated: 21/Aug/13 00:08
• Resolved: 02/Aug/13 04:44

Description

I was running the following test over a Distributed Cluster:
bin/hbase org.apache.hadoop.hbase.IntegrationTestsDriver IntegrationTestDataIngestSlowDeterministic

The IntegrationTestingUtility.restoreCluster() is called in the teardown phase of the test.
For a distributed cluster, it ends up calling DistributedHBaseCluster.restoreClusterStatus, which does the task
of restoring the cluster back to original state.

The restore steps done here, does not solve one specific case:
When the initial HBase Master is currently down, and the current HBase Master is different from the initial one.

You get into this flow:

//check whether current master has changed
if (!ServerName.isSameHostnameAndPort(initial.getMaster(), current.getMaster()))

{ ............. }

In the above code path, the current backup masters are stopped, and the current active master is also stopped.
At this point, for the aforementioned usecase, none of the Hbase Masters would be available, hence the subsequent
attempts to do any operation over the cluster would fail, resulting in Test Failure.

Issue Links

• is a clone of: HBASE-8928 Make ChaosMonkey & LoadTest tools extensible, to allow addition of more actions and policies.