[HBASE-26468] Region Server doesn't exit cleanly incase it crashes.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 2.5.0, 3.0.0-alpha-2, 1.7.2, 2.4.9
• Component/s: regionserver
• Hadoop Flags: Reviewed
• Assignee: Rushabh Shah
• Reporter: Rushabh Shah
• Created: 18/Nov/21 21:59
• Updated: 02/Dec/21 05:38
• Resolved: 01/Dec/21 05:51

Description

Observed this in our production cluster running 1.6 version.
RS crashed due to some reason but the process was still running. On debugging more, found out there was 1 non-daemon thread running and that was not allowing RS to exit cleanly. Our clusters are managed by Ambari and have auto restart capability within them. But since the process was running and pid file was present, Ambari also couldn't do much. There will be some bug where we will miss to stop some non daemon thread. Shutdown hook will not be called unless one of the following 2 conditions are met:

1. The Java virtual machine shuts down in response to two kinds of events:
   The program exits normally, when the last non-daemon thread exits or when the exit (equivalently, System.exit) method is invoked, or
2. The virtual machine is terminated in response to a user interrupt, such as typing ^C, or a system-wide event, such as user logoff or system shutdown.

Considering the first condition, when the last non-daemon thread exits or when the exit method is invoked.

Below is the code snippet from HRegionServerCommandLine.java

  private int start() throws Exception {
    try {
      if (LocalHBaseCluster.isLocal(conf)) {
         // Ignore this.
      } else {
        HRegionServer hrs = HRegionServer.constructRegionServer(regionServerClass, conf);
        hrs.start();
        hrs.join();
        if (hrs.isAborted()) {
          throw new RuntimeException("HRegionServer Aborted");
        }
      }
    } catch (Throwable t) {
      LOG.error("Region server exiting", t);
      return 1;
    }
    return 0;
  }

Within HRegionServer, there is a subtle difference between when a server is aborted v/s when it is stopped. If it is stopped, then isAborted will return false and it will exit with return code 0.

Below is the code from ServerCommandLine.java

  public void doMain(String args[]) {
    try {
      int ret = ToolRunner.run(HBaseConfiguration.create(), this, args);
      if (ret != 0) {
        System.exit(ret);
      }
    } catch (Exception e) {
      LOG.error("Failed to run", e);
      System.exit(-1);
    }
  }

If return code is 0, then it won't call System.exit. This means JVM will wait to call ShutdownHook until all non daemon threads are stopped which means infinite wait if we don't close all non-daemon threads cleanly.

Issue Links

• links to: GitHub Pull Request #3862
• links to: GitHub Pull Request #3865
• links to: GitHub Pull Request #3866