Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16014

HiveMetastoreChecker should use hive.metastore.fshandler.threads instead of hive.mv.files.thread for pool size

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.3.0
    • None
    • None

    Description

      HiveMetastoreChecker uses hive.mv.files.thread configuration value for determining the pool size as below :

      private void checkPartitionDirs(Path basePath, Set<Path> allDirs, int maxDepth) throws IOException, HiveException {
          ConcurrentLinkedQueue<Path> basePaths = new ConcurrentLinkedQueue<>();
          basePaths.add(basePath);
          Set<Path> dirSet = Collections.newSetFromMap(new ConcurrentHashMap<Path, Boolean>());
          // Here we just reuse the THREAD_COUNT configuration for
          // HIVE_MOVE_FILES_THREAD_COUNT
          int poolSize = conf.getInt(ConfVars.HIVE_MOVE_FILES_THREAD_COUNT.varname, 15);
      
          // Check if too low config is provided for move files. 2x CPU is reasonable max count.
          poolSize = poolSize == 0 ? poolSize : Math.max(poolSize,
              Runtime.getRuntime().availableProcessors() * 2);
      

      msck is commonly used to add the missing partitions for the table from the Filesystem. In such a case different pool sizes for HMSHandler and HiveMetastoreChecker can affect the performance. Eg. If hive.metastore.fshandler.threads is set to a lower value like 15 and hive.mv.files.thread is much higher like 100 or vice versa the smaller pool will become the bottleneck. If would be good to use hive.metastore.fshandler.threads to size the pool for HiveMetastoreChecker since the number missing partitions and number of partitions to be added will most likely be the same. In such a case the performance of the query will be optimum when both the pool sizes are same.

      Since it is possible to tune both the configs individually it will be very likely that they may be different. But since there is a strong co-relation between amount of work done by HiveMetastoreChecker and HiveMetastore.add_partitions call it might be a good idea to use hive.metastore.fshandler.threads for pool size instead of hive.mv.files.thread

      Attachments

        1. HIVE-16014.01.patch
          1 kB
          Vihang Karajgaonkar
        2. HIVE-16014.02.patch
          2 kB
          Vihang Karajgaonkar
        3. HIVE-16014.03.patch
          2 kB
          Vihang Karajgaonkar

        Issue Links

          Activity

            People

              vihangk1 Vihang Karajgaonkar
              vihangk1 Vihang Karajgaonkar
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: