Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14925

MSCK repair table hang while running with multi threading enabled

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 2.2.0
    • Fix Version/s: 3.2.0
    • Component/s: CLI
    • Labels:
      None
    • Release Note:
      Hide
      Issue: MSCK is failing in multithreaded execution

      Solution:
        - Moved Path processor logic to an external class which will avoid code duplication and it will be used in both multi-threaded and single threaded execution.
      Show
      Issue: MSCK is failing in multithreaded execution Solution:   - Moved Path processor logic to an external class which will avoid code duplication and it will be used in both multi-threaded and single threaded execution.

      Description

      MSCK REPAIR TABLE hanging while running with multi-threading enabled (default). I think it is because of a major design flaw in how thread pool implemented in HiveMetaSoreChecker class / checkPartitionDirs method. This method has a thread pool which register Callable but callable makes a recursive call to checkPartitionDirs method again. This code will hang when number of directories is more than thread pool size.

        Attachments

        1. HIVE-14925.patch
          8 kB
          Ratheesh Kamoor

          Activity

            People

            • Assignee:
              rkamoor Ratheesh Kamoor
              Reporter:
              rkamoor Ratheesh Kamoor
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated: