Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-15807

MSCK operation hangs in HiveMetaStoreChecker.checkPartitionDirs

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 2.2.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The seems to be a regression from HIVE-14511. The operation was hung in checkPartitionDirs. The data has 3 levels of partitions (month, date, id) which has total of 800 partitions.

      An example path would look like month=9/day=30/id=12

      The default value for hive config hive.mv.files.thread was set to 128. I have attached the jstack of hive process used to run msck command

      checkPartitionDirs is implemented as recursive function which uses same pool to submit worker threads. It seems thread pool ran out of thread to do the actual work and all threads seems to be waiting and hung. Please take a look the stack and confirm if this is the case here.

        Attachments

        1. msck-jstack.txt
          157 kB
          Subramanyam Pattipaka

          Issue Links

            Activity

              People

              • Assignee:
                pxiong Pengcheng Xiong
                Reporter:
                pattipaka Subramanyam Pattipaka
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: