Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-8699 Enable support for common map join [Spark Branch]
  3. HIVE-8943

Fix memory limit check for combine nested mapjoins [Spark Branch]

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • spark-branch
    • 1.1.0
    • Spark
    • None

    Description

      Its the opposite problem of what we thought in HIVE-8701.

      SparkMapJoinOptimizer does combine nested mapjoins into one work due to removal of RS for big-table. So we need to enhance the check to calculate if all the MapJoins in that work (spark-stage) will fit into the memory, otherwise it might overwhelm memory for that particular spark executor.

      Attachments

        1. HIVE-8943.1-spark.patch
          46 kB
          Szehon Ho
        2. HIVE-8943.1-spark.patch
          46 kB
          Szehon Ho
        3. HIVE-8943.2-spark.patch
          47 kB
          Szehon Ho
        4. HIVE-8943.3-spark.patch
          84 kB
          Szehon Ho
        5. HIVE-8943.4-spark.patch
          82 kB
          Szehon Ho
        6. HIVE-8943.5-spark.patch
          82 kB
          Szehon Ho
        7. HIVE-8943.6-spark.patch
          82 kB
          Szehon Ho
        8. HIVE-8943.7-spark.patch
          82 kB
          Szehon Ho

        Issue Links

          Activity

            People

              szehon Szehon Ho
              szehon Szehon Ho
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: