[HBASE-19533] How to do controlled shutdown in branch-2?

• Type: Task
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Fix Version/s: 3.0.0-beta-2
• Assignee: Unassigned
• Reporter: Michael Stack
• Created: 16/Dec/17 21:44
• Updated: 16/Dec/23 09:45

Description

Before HBASE-18946, setting shutdown of a cluster, the Master would exit immediately. RegionServers would run region closes and then try and notify the Master of the close and would spew exceptions that the Master was unreachable.

This is different to how branch-1 used to do it. It used to keep Master up and it would be like the captain of the ship, the last to go down. As of HBASE-18946, this is again the case but there are still open issues.

1. Usually Master does all open and close of regions. On cluster shutdown, it is the one time where the Regions run the region close. Currently, the regions report the close to the Master which disregards the message since it did not start the region closes. Should we do different? Try and update state in hbase:meta setting it to CLOSE? We might not be able to write CLOSE for all regions since hbase:meta will be closing too (the RS that is hosting hbase:meta will close it last.... but that may not be enough).
2. Should the Master run the cluster shutdown sending out close for all regions? What if cluster of 1M regions? Untenable? Send a message per server? That might be better.

Anyways, this needs attention. Filing issue in meantime.

Sub-Tasks

• Make ExecutorService threads daemon=true.: Resolved
• Signalling server-hosted-clients to abort retries: Resolved