Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-22411

Performance degradation on single row inserts

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0.0
    • Component/s: Hive, Transactions
    • Labels:
      None

      Description

      Executing single insert statements on a transactional table effects write performance on a s3 file system. Each insert creates a new delta directory. After each insert hive calculates statistics like number of file in the table and total size of the table. In order to calculate these, it traverses the directory recursively. During the recursion for each path a separateĀ listStatus call is executed. In the end the more delta directory you have the more time it takes to calculate the statistics.

      Therefore insertion time goes up linearly:

      The fix is to useĀ fs.listFiles(path, /*recursive*/ true) instead the handcrafter recursive method/

        Attachments

        1. HIVE-22411.2.patch
          5 kB
          Attila Magyar
        2. HIVE-22411.3.patch
          13 kB
          Attila Magyar
        3. Screen Shot 2019-10-17 at 8.40.50 PM.png
          58 kB
          Attila Magyar

          Issue Links

            Activity

              People

              • Assignee:
                amagyar Attila Magyar
                Reporter:
                amagyar Attila Magyar
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: