Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-15682

Eliminate per-row based dummy iterator creation

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.2.0
    • 2.3.0
    • Spark
    • None

    Description

      HIVE-15580 introduced a dummy iterator per input row which can be eliminated. This is because SparkReduceRecordHandler is able to handle single key value pairs. We can refactor this part of code 1. to remove the need for a iterator and 2. to optimize the code path for per (key, value) based (instead of (key, value iterator)) processing. It would be also great if we can measure the performance after the optimizations and compare to performance prior to HIVE-15580.

      Attachments

        1. HIVE-15682.patch
          2 kB
          Xuefu Zhang

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-15580 Eliminate unbounded memory usage for orderBy and groupBy in Hive on Spark

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-15683 Make what's done in HIVE-15580 for group by configurable

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              xuefuz Xuefu Zhang
              xuefuz Xuefu Zhang
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: