[HBASE-12426] Dead Region Servers after Decommissioning

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 0.96.1.1
• Component/s: Balancer

Environment

RHEL 6.5

• Assignee: Unassigned
• Reporter: Nishanth Shajahan
• Created: 04/Nov/14 19:22
• Updated: 17/Jun/22 18:20
• Resolved: 17/Jun/22 18:20

Description

I initially had a set of 5 region servers which had a single table which was pre split into 30 regions and was evenly distributed to all the regions with data.I then went ahead and removed/decommissioned a coupe of region servers,so in the end I have 3 region servers.Ran hbase hbck and verified there were 0 inconsistencies.However when 'status' command is issued is from hbase shell it shows a dead region server and the same is displayed in master UI as well.Fail over of hbase master did not fix the issue.On investigation we could see some WAL entries which was still pointing to the old region server.
/hbase/WALs/myserver,60020,1406745344969-splitting

After removing these orphan entries from hdfs and master failover the dead region servers went away.I wonder if this could have caused any replication issues in the cluster.