Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-8118 Support work that have multiple child works to work around SPARK-3622 [Spark Branch]
  3. HIVE-8436

Modify SparkWork to split works with multiple child works [Spark Branch]

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.1.0
    • Spark
    • None

    Description

      Based on the design doc, we need to split the operator tree of a work in SparkWork if the work is connected to multiple child works. The way splitting the operator tree is performed by cloning the original work and removing unwanted branches in the operator tree. Please refer to the design doc for details.

      This process should be done right before we generate SparkPlan. We should have a utility method that takes the orignal SparkWork and return a modified SparkWork.

      This process should also keep the information about the original work and its clones. Such information will be needed during SparkPlan generation (HIVE-8437).

      Attachments

        1. HIVE-8436.10-spark.patch
          603 kB
          Chao Sun
        2. HIVE-8436.11-spark.patch
          605 kB
          Chao Sun
        3. HIVE-8436.1-spark.patch
          503 kB
          Chao Sun
        4. HIVE-8436.2-spark.patch
          502 kB
          Chao Sun
        5. HIVE-8436.3-spark.patch
          495 kB
          Chao Sun
        6. HIVE-8436.4-spark.patch
          513 kB
          Chao Sun
        7. HIVE-8436.5-spark.patch
          513 kB
          Chao Sun
        8. HIVE-8436.6-spark.patch
          532 kB
          Chao Sun
        9. HIVE-8436.7-spark.patch
          533 kB
          Chao Sun
        10. HIVE-8436.8-spark.patch
          533 kB
          Chao Sun
        11. HIVE-8436.9-spark.patch
          603 kB
          Chao Sun

        Issue Links

          Activity

            People

              csun Chao Sun
              xuefuz Xuefu Zhang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: