[HBASE-23169] Random region server aborts while clearing Old Wals

• Type: Bug
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Affects Version/s: 1.4.10, 1.4.11
• Component/s: regionserver, Replication, wal
• Labels: patch
• Flags: Patch
• External issue URL: https://jira.apache.org/jira/browse/HBASE-22784
• External issue ID: HBASE-22784
• Assignee: Wellington Chevreuil
• Reporter: Karthick
• Created: 14/Oct/19 02:51
• Updated: 25/Oct/19 05:09

Description

After applying the patch given in HBASE-22784 random region server aborts were noticed. This happens in ReplicationResourceShipper thread while writing the replication wal position.

2019-10-05 08:17:28,132 FATAL [regionserver//172.20.20.20:16020.replicationSource.172.20.20.20%2C16020%2C1570193969775,2] regionserver.HRegionServer: ABORTING region server 172.20.20.20,16020,1570193969775: Failed to write replication wal position (filename=172.20.20.20%2C16020%2C1570193969775.1570288637045, position=127494739)2019-10-05 08:17:28,132 FATAL [regionserver//172.20.20.20:16020.replicationSource.172.20.20.20%2C16020%2C1570193969775,2] regionserver.HRegionServer: ABORTING region server 172.20.20.20,16020,1570193969775: Failed to write replication wal position (filename=172.20.20.20%2C16020%2C1570193969775.1570288637045, position=127494739)org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/replication/rs/172.20.20.20,16020,1570193969775/2/172.20.20.20%2C16020%2C1570193969775.1570288637045 at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1327) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:422) at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:824) at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:874) at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:868) at org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.setLogPosition(ReplicationQueuesZKImpl.java:155) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:194) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceShipperThread.updateLogPosition(ReplicationSource.java:727) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceShipperThread.shipEdits(ReplicationSource.java:698) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceShipperThread.run(ReplicationSource.java:551)2019-10-05 08:17:28,133 FATAL [regionserver//172.20.20.20:16020.replicationSource.172.20.20.20%2C16020%2C1570193969775,2] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint

Issue Links

• is related to: HBASE-23205 Correctly update the position of WALs currently being replicated.
• relates to: HBASE-22784 OldWALs not cleared in a replication slave cluster (cyclic replication bw 2 clusters)
• relates to: HBASE-23205 Correctly update the position of WALs currently being replicated.