[HBASE-14261] Enhance Chaos Monkey framework by adding zookeeper and datanode fault injections.

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0, 1.3.0, 0.98.15, 1.0.3, 1.1.3, 2.0.0
• Component/s: integration tests
• Hadoop Flags: Reviewed

Release Note

This change augments existing chaos monkey framework with actions for restarting underlying zookeeper quorum and hdfs nodes of distributed hbase cluster. One assumption made while creating zk actions are that zookeper ensemble is an independent external service and won't be managed by hbase cluster. For these actions to work as expected, the following parameters need to be configured appropriately.

{code}
<property>
  <name>hbase.it.clustermanager.hadoop.home</name>
  <value>$HADOOP_HOME</value>
</property>
<property>
  <name>hbase.it.clustermanager.zookeeper.home</name>
  <value>$ZOOKEEPER_HOME</value>
</property>
<property>
  <name>hbase.it.clustermanager.hbase.user</name>
  <value>hbase</value>
</property>
<property>
  <name>hbase.it.clustermanager.hadoop.hdfs.user</name>
  <value>hdfs</value>
</property>
<property>
  <name>hbase.it.clustermanager.zookeeper.user</name>
  <value>zookeeper</value>
</property>
{code}

The service user related configurations are newly introduced since in prod/test environments each service is managed by different user. Once the above parameters are configured properly, you can start using them as needed. An example usage for invoking these new actions is:

{{./hbase org.apache.hadoop.hbase.IntegrationTestAcidGuarantees -m serverAndDependenciesKilling}}

• Assignee: Srikanth Srungarapu
• Reporter: Srikanth Srungarapu
• Created: 19/Aug/15 23:30
• Updated: 27/Jan/16 15:28
• Resolved: 03/Sep/15 22:22

Description

One of the shortcomings of existing ChaosMonkey framework is lack of fault injections for hbase dependencies like zookeeper, hdfs etc. This patch attempts to solve this problem partially by adding datanode and zk node fault injections.