[HBASE-25032] Do not assign regions to region server which has not called regionServerReport yet - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 3.0.0-alpha-1, 2.5.0
Component/s: None
Labels:
- master
- regionserver

Hadoop Flags:

Reviewed
Release Note:

Hide


After this change a region server can only accept regions (as seen by master) after it's first report to master is sent successfully. Prior to this change there could be cases where the region server finishes calling regionServerStartup but is actually stuck during initialization due to issues like misconfiguration and master tries to assign regions and they are stuck because the region server is in a weird state and not ready to serve them.

Show
 After this change a region server can only accept regions (as seen by master) after it's first report to master is sent successfully. Prior to this change there could be cases where the region server finishes calling regionServerStartup but is actually stuck during initialization due to issues like misconfiguration and master tries to assign regions and they are stuck because the region server is in a weird state and not ready to serve them.

Description

As part of RS start up, RS reports for duty to Master . Master acknowledges the request and adds it to the onlineServers list for further assigning any regions to the RS

Once Master acknowledges the reportForDuty and sends back the response, RS does a bunch of stuff like initializing replication sources etc before becoming online. However, sometimes there could be an issue with initializing replication sources when it is unable to connect to peer clusters because of some kerberos configuration and there would be a delay of around 20 mins in becoming online.

Since master considers it online, it tries to assign regions and which fails with ServerNotRunningYet exception, then the master tries to unassign which again fails with the same exception leading the region to FAILED_CLOSE state.

It would be good to have a check to see if the RS is ready to accept the assignment requests before adding it to online servers list which would account for any such delays as described above