Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21071

Improve getInputSummary

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0, 4.0.0
    • 4.0.0, 4.0.0-alpha-1
    • HiveServer2
    • None
    • Patch

    Description

      There is a global lock in the getInptSummary code, so it is important that it be fast. The current implementation has quite a bit of overhead that can be re-engineered.

      For example, the current implementation keeps a map of File Path to ContentSummary object. This map is populated by several threads concurrently. The method then loops through the map, in a single thread, at the end to add up all of the ContentSummary objects and ignores the paths. The code can be be re-engineered to not use a map, or a collection at all, to store the results and instead just keep a running tally. By keeping a tally, there is no O(n) operation at the end to perform the addition.

      There are other things can be improved. The method returns an object which is never used anywhere, so change method to void return type.

      Attachments

        1. HIVE-21071.1.patch
          15 kB
          David Mollitor
        2. HIVE-21071.2.patch
          13 kB
          David Mollitor
        3. HIVE-21071.3.patch
          13 kB
          David Mollitor
        4. HIVE-21071.4.patch
          13 kB
          David Mollitor
        5. HIVE-21071.5.patch
          13 kB
          David Mollitor
        6. HIVE-21071.6.patch
          12 kB
          David Mollitor
        7. HIVE-21071.7.patch
          40 kB
          David Mollitor
        8. HIVE-21071.8.patch
          40 kB
          David Mollitor
        9. HIVE-21071.9.patch
          40 kB
          David Mollitor
        10. HIVE-21071.10.patch
          40 kB
          David Mollitor
        11. HIVE-21071.11.patch
          42 kB
          David Mollitor

        Issue Links

          Activity

            People

              belugabehr David Mollitor
              belugabehr David Mollitor
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: