[HBASE-16367] Race between master and region server initialization may lead to premature server abort

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.2
• Fix Version/s: 1.4.0, 2.0.0
• Hadoop Flags: Reviewed
• Assignee: Ted Yu
• Reporter: Ted Yu
• Created: 06/Aug/16 09:20
• Updated: 09/Jan/18 19:24
• Resolved: 08/Aug/16 17:42

Description

I was troubleshooting a case where hbase (1.1.2) master always dies shortly after start - see attached master log snippet.

It turned out that master initialization thread was racing with HRegionServer#preRegistrationInitialization() (initializeZooKeeper, actually) since HMaster extends HRegionServer.
Through additional logging in master:

    this.oldLogDir = createInitialFileSystemLayout();
    HFileSystem.addLocationsOrderInterceptor(conf);
    LOG.info("creating splitLogManager");

I found that execution didn't reach the last log line before region server declared cluster Id being null.

Issue Links

• is related to: HBASE-19694 The initialization order for a fresh cluster is incorrect
• is related to: HBASE-17902 Backport HBASE-16367 "Race between master and region server initialization may lead to premature server abort" to 1.3