Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16385

StatsNoJobTask could exit early before all partitions have been processed

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0, 2.1.0
    • Fix Version/s: 2.3.0, 3.0.0
    • Component/s: Statistics
    • Labels:
      None

      Description

      For a partitioned table, the class StatsNoJobTask is supposed to launch threads for all partitions and compute their stats. However, it could exit early after at most 100 seconds:

        private void shutdownAndAwaitTermination(ExecutorService threadPool) {
      
          // Disable new tasks from being submitted
          threadPool.shutdown();
          try {
      
            // Wait a while for existing tasks to terminate
            if (!threadPool.awaitTermination(100, TimeUnit.SECONDS)) {
              // Cancel currently executing tasks
              threadPool.shutdownNow();
      
              // Wait a while for tasks to respond to being cancelled
              if (!threadPool.awaitTermination(100, TimeUnit.SECONDS)) {
                LOG.debug("Stats collection thread pool did not terminate");
              }
            }
          } catch (InterruptedException ie) {
      
            // Cancel again if current thread also interrupted
            threadPool.shutdownNow();
      
            // Preserve interrupt status
            Thread.currentThread().interrupt();
          }
        }
      

      The shutdown call does not wait for all submitted tasks to complete, and the awaitTermination call waits at most 100 seconds.

        Attachments

        1. HIVE-16385.1.patch
          1 kB
          Chao Sun

          Activity

            People

            • Assignee:
              csun Chao Sun
              Reporter:
              csun Chao Sun
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: