Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-8649

Increase level of parallelism in reduce phase [Spark Branch]

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.0
    • Component/s: Spark
    • Labels:
      None

      Description

      We calculate the number of reducers based on the same code for MapReduce. However, reducers are vastly cheaper in Spark and it's generally recommended we have many more reducers than in MR.

      Sandy Ryza who works on Spark has some ideas about a heuristic.

        Attachments

        1. HIVE-8649.2-spark.patch
          6 kB
          Jimmy Xiang
        2. HIVE-8649.1-spark.patch
          5 kB
          Jimmy Xiang

          Activity

            People

            • Assignee:
              jxiang Jimmy Xiang
              Reporter:
              brocknoland Brock Noland
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: