[HBASE-25902] Add missing CFs in meta during HBase 1 to 2.3+ Upgrade - ASF JIRA

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Critical
Resolution: Fixed
Affects Version/s: 2.3.0, 2.4.0
Fix Version/s: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.5
Component/s: meta, Operability
Labels:
None

Hadoop Flags:

Reviewed
Release Note:

Hide
While upgrading cluster from 1.x to 2.3+ versions, after the active master is done setting it's status as 'Initialized', it attempts to add 'table' and 'repl_barrier' CFs in meta. Once CFs are added successfully, master is aborted with PleaseRestartMasterException because master has missed certain initialization events (e.g ClusterSchemaService is not initialized and tableStateManager fails to migrate table states from ZK to meta due to missing CFs). Subsequent active master initialization is expected to be smooth.
In the presence of multi masters, when one of them becomes active for the first time after upgrading to HBase 2.3+, it is aborted after fixing CFs in meta and one of the other backup masters will take over and become active soon. Hence, overall this is expected to be smooth upgrade if we have backup masters configured. If not, operator is expected to restart same master again manually.

Show
While upgrading cluster from 1.x to 2.3+ versions, after the active master is done setting it's status as 'Initialized', it attempts to add 'table' and 'repl_barrier' CFs in meta. Once CFs are added successfully, master is aborted with PleaseRestartMasterException because master has missed certain initialization events (e.g ClusterSchemaService is not initialized and tableStateManager fails to migrate table states from ZK to meta due to missing CFs). Subsequent active master initialization is expected to be smooth. In the presence of multi masters, when one of them becomes active for the first time after upgrading to HBase 2.3+, it is aborted after fixing CFs in meta and one of the other backup masters will take over and become active soon. Hence, overall this is expected to be smooth upgrade if we have backup masters configured. If not, operator is expected to restart same master again manually.

Description

Making note of this issue in case others run into it. At my place of employ, we tried to upgrade a cluster that was an hbase-1.2.x version to an hbase-2.3.5 but it failed because meta didn't have the 'table' column family.

Up to 2.3.0, hbase:meta was hardcoded. ~~HBASE-12035~~ added the 'table' CF for hbase-2.0.0. ~~HBASE-23782~~ (2.3.0) undid hardcoding of the hbase:meta schema; i.e. reading hbase:meta schema from the filesystem. The hbase:meta schema is only created on initial install. If an upgrade over existing data, the hbase-1 hbase:meta will not be suitable for hbase-2.3.x context as it will be missing columnfamilies needed to run (~~HBASE-23055~~ made it so hbase:meta could be altered (2.3.0) but probably of no use since Master won't come up).

It would be a nice-to-have if a user could go from hbase1 to hbase.2.3.0 w/o having to first install an hbase2 that is earlier than 2.3.0 but needs to be demand before we would work on it; meantime, install an intermediate hbase2 version before going to hbase-2.3.0+ if coming from hbase-1.x