[HBASE-22923] hbase:meta is assigned to localhost when we downgrade the hbase version

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.8
• Fix Version/s: 3.0.0-alpha-1, 2.5.0, 2.3.6, 1.7.1, 2.4.5
• Hadoop Flags: Reviewed

Release Note

Introduced new config: hbase.min.version.move.system.tables

When the operator uses this configuration option, any version between
the current cluster version and the value of "hbase.min.version.move.system.tables"
does not trigger any auto-region movement. Auto-region movement here
refers to auto-migration of system table regions to newer server versions.
It is assumed that the configured range of versions does not require special
handling of moving system table regions to higher versioned RegionServer.
This auto-migration is done by AssignmentManager#checkIfShouldMoveSystemRegionAsync().
Example: Let's assume the cluster is on version 1.4.0 and we have
set "hbase.min.version.move.system.tables" as "2.0.0". Now if we upgrade
one RegionServer on 1.4.0 cluster to 1.6.0 (< 2.0.0), then AssignmentManager will
not move hbase:meta, hbase:namespace and other system table regions
to newly brought up RegionServer 1.6.0 as part of auto-migration.
However, if we upgrade one RegionServer on 1.4.0 cluster to 2.2.0 (> 2.0.0),
then AssignmentManager will move all system table regions to newly brought
up RegionServer 2.2.0 as part of auto-migration done by
AssignmentManager#checkIfShouldMoveSystemRegionAsync().

Overall, assuming we have system RSGroup where we keep HBase system tables, if we use
config "hbase.min.version.move.system.tables" with value x.y.z then while upgrading cluster to
version greater than or equal to x.y.z, the first RegionServer that we upgrade must
belong to system RSGroup only.

• Assignee: Viraj Jasani
• Reporter: wenbang
• Created: 26/Aug/19 08:23
• Updated: 05/Jul/21 08:27
• Resolved: 03/Jul/21 17:18

Description

When we downgrade the hbase version（rsgroup enable）, we found that the hbase:meta table could not be assigned.

master.AssignmentManager: Failed assignment of hbase:meta,,1.1588230740 to localhost,1,1, trying to assign elsewhere instead; try=1 of 10 java.io.IOException: Call to localhost/127.0.0.1:1 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: localhost/127.0.0.1:1

hbase group list:

  HBASE_META group（hbase:meta and other system tables）

  default group

1.Down grade all servers in HBASE_META first

2.higher version servers is in default

3.hbase:meta assigned to localhost

For system table, we assign them to a server with highest version.

AssignmentManager#getExcludedServersForSystemTable

But did not consider the rsgroup.

Issue Links

• links to: GitHub Pull Request #3438
• links to: GitHub Pull Request #3439
• links to: GitHub Pull Request #3455
• links to: GitHub Pull Request #3456