[HBASE-27192] The retry number for TestSeparateClientZKCluster is too small

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.5.0, 3.0.0-alpha-4, 2.4.14
• Component/s: test, Zookeeper
• Hadoop Flags: Reviewed
• Assignee: Duo Zhang
• Reporter: Duo Zhang
• Created: 12/Jul/22 06:33
• Updated: 15/Jul/22 12:18
• Resolved: 14/Jul/22 00:57

Description

The retry number is only 2, checking the log output, we will fail the request within 600ms, which is too small in testMetaMoveDuringClientZkClusterRestart, as in this method we will shutdown the client zookeeper, the retry interval when updating zookeeper is way more greater, usually several seconds. For example

2022-07-11T00:51:09,998 DEBUG [ClientZKUpdater-/hbase/meta-region-server] zookeeper.RecoverableZooKeeper(303): Retry, connectivity issue (JVM Pause?); quorum=localhost:21828,exceptionorg.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/meta-region-server=
2022-07-11T00:51:11,187 DEBUG [ClientZKUpdater-/hbase/meta-region-server] zookeeper.RecoverableZooKeeper(303): Retry, connectivity issue (JVM Pause?); quorum=localhost:21828,exceptionorg.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/meta-region-server=
2022-07-11T00:51:13,617 WARN  [HBase-Metrics2-1] impl.MetricsConfig(136): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2022-07-11T00:51:13,852 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta'
2022-07-11T00:51:13,853 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testAsyncTable'
2022-07-11T00:51:13,854 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testMetaMoveDuringClientZkClusterRestart'
2022-07-11T00:51:14,124 ERROR [ClientZKUpdater-/hbase/meta-region-server] zookeeper.RecoverableZooKeeper(300): ZooKeeper setData failed after 2 attempts
2022-07-11T00:51:14,124 DEBUG [ClientZKUpdater-/hbase/meta-region-server] zksyncer.ClientZKSyncer(179): Failed to set data for /hbase/meta-region-server to client ZK, will retry later
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/meta-region-server
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) ~[zookeeper-3.5.7.jar:3.5.7]
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.5.7.jar:3.5.7]
	at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:2384) ~[zookeeper-3.5.7.jar:3.5.7]
	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:428) ~[hbase-zookeeper-3.0.0-alpha-4-SNAPSHOT.jar:3.0.0-alpha-4-SNAPSHOT]
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:558) ~[hbase-zookeeper-3.0.0-alpha-4-SNAPSHOT.jar:3.0.0-alpha-4-SNAPSHOT]
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:603) ~[hbase-zookeeper-3.0.0-alpha-4-SNAPSHOT.jar:3.0.0-alpha-4-SNAPSHOT]
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:597) ~[hbase-zookeeper-3.0.0-alpha-4-SNAPSHOT.jar:3.0.0-alpha-4-SNAPSHOT]
	at org.apache.hadoop.hbase.master.zksyncer.ClientZKSyncer.setDataForClientZkUntilSuccess(ClientZKSyncer.java:175) ~[classes/:?]
	at org.apache.hadoop.hbase.master.zksyncer.ClientZKSyncer.access$300(ClientZKSyncer.java:45) ~[classes/:?]
	at org.apache.hadoop.hbase.master.zksyncer.ClientZKSyncer$ClientZkUpdater.run(ClientZKSyncer.java:319) ~[classes/:?]

The first retry log is at 09.998, the second one is at 11.187, and the third one is at 14.124.

Let's just remove the line which set retry number to 2.

Issue Links

• links to: GitHub Pull Request #4614