Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-15682

Eliminate per-row based dummy iterator creation

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.3.0
    • Component/s: Spark
    • Labels:
      None

      Description

      HIVE-15580 introduced a dummy iterator per input row which can be eliminated. This is because SparkReduceRecordHandler is able to handle single key value pairs. We can refactor this part of code 1. to remove the need for a iterator and 2. to optimize the code path for per (key, value) based (instead of (key, value iterator)) processing. It would be also great if we can measure the performance after the optimizations and compare to performance prior to HIVE-15580.

        Attachments

        1. HIVE-15682.patch
          2 kB
          Xuefu Zhang

          Issue Links

            Activity

              People

              • Assignee:
                xuefuz Xuefu Zhang
                Reporter:
                xuefuz Xuefu Zhang
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: