[HBASE-10367] RegionServer graceful stop / decommissioning

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0-alpha-4, 2.0.0
• Hadoop Flags: Reviewed

Release Note

Added three top level Admin APIs to help decommissioning and graceful stop of region servers.

  /**
   * Mark region server(s) as decommissioned to prevent additional regions from getting
   * assigned to them. Optionally unload the regions on the servers. If there are multiple servers
   * to be decommissioned, decommissioning them at the same time can prevent wasteful region
   * movements. Region unloading is asynchronous.
   * @param servers The list of servers to decommission.
   * @param offload True to offload the regions from the decommissioned servers
   */
  void decommissionRegionServers(List<ServerName> servers, boolean offload) throws IOException;

  /**
   * List region servers marked as decommissioned, which can not be assigned regions.
   * @return List of decommissioned region servers.
   */
  List<ServerName> listDecommissionedRegionServers() throws IOException;

  /**
   * Remove decommission marker from a region server to allow regions assignments.
   * Load regions onto the server if a list of regions is given. Region loading is
   * asynchronous.
   * @param server The server to recommission.
   * @param encodedRegionNames Regions to load onto the server.
   */
  void recommissionRegionServer(ServerName server, List<byte[]> encodedRegionNames) throws IOException;

• Assignee: Jerry He
• Reporter: Enis Soztutar
• Created: 16/Jan/14 23:03
• Updated: 17/Jun/22 05:00
• Resolved: 20/Oct/17 05:17

Description

Right now, we have a weird way of node decommissioning / graceful stop, which is a graceful_stop.sh bash script, and a region_mover ruby script, and some draining server support which you have to manually write to a znode (really!). Also draining servers is only partially supported in LB operations (LB does take that into account for roundRobin assignment, but not for normal balance)
See
http://hbase.apache.org/book/node.management.html and HBASE-3071

I think we should support graceful stop as a first class citizen. Thinking about it, it seems that the difference between regionserver stop and graceful stop is that regionserver stop will close the regions, but the master will only assign them after the znode is deleted.

In the new master design (or even before), if we allow RS to be able to close regions on its own (without master initiating it), then graceful stop becomes regular stop. The RS already closes the regions cleanly, and will reject new region assignments, so that we don't need much of the balancer or draining server trickery.

This ties into the new master/AM redesign (HBASE-5487), but still deserves it's own jira. Let's use this to brainstorm on the design.

Issue Links

• incorporates: HBASE-17369 Add ACL to the new region server drain related API
• is related to: HBASE-6402 Ability to explicitly include and exclude hosts from cluster
• relates to: HBASE-17521 Avoid stopping the load balancer in graceful stop
• relates to: HBASE-16010 Put draining function through Admin API

Sub-Tasks

• Add a draining_node script to manage nodes in draining mode: Closed
• Make sure all balancer actions respect decommissioned server: Closed