[HIVE-16385] StatsNoJobTask could exit early before all partitions have been processed

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0, 2.1.0
• Fix Version/s: 2.3.0, 3.0.0
• Component/s: Statistics
• Assignee: Chao Sun
• Reporter: Chao Sun
• Created: 05/Apr/17 18:06
• Updated: 23/May/18 00:00
• Resolved: 07/Apr/17 22:48

Description

For a partitioned table, the class StatsNoJobTask is supposed to launch threads for all partitions and compute their stats. However, it could exit early after at most 100 seconds:

  private void shutdownAndAwaitTermination(ExecutorService threadPool) {

    // Disable new tasks from being submitted
    threadPool.shutdown();
    try {

      // Wait a while for existing tasks to terminate
      if (!threadPool.awaitTermination(100, TimeUnit.SECONDS)) {
        // Cancel currently executing tasks
        threadPool.shutdownNow();

        // Wait a while for tasks to respond to being cancelled
        if (!threadPool.awaitTermination(100, TimeUnit.SECONDS)) {
          LOG.debug("Stats collection thread pool did not terminate");
        }
      }
    } catch (InterruptedException ie) {

      // Cancel again if current thread also interrupted
      threadPool.shutdownNow();

      // Preserve interrupt status
      Thread.currentThread().interrupt();
    }
  }

The shutdown call does not wait for all submitted tasks to complete, and the awaitTermination call waits at most 100 seconds.