Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16923 Hive-on-Spark DPP Improvements
  3. HIVE-17405

HoS DPP ConstantPropagate should use ConstantPropagateOption.SHORTCUT

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0
    • Component/s: Spark
    • Labels:
      None
    • Target Version/s:

      Description

      In SparkCompiler#runDynamicPartitionPruning we should change new ConstantPropagate().transform(parseContext) to new ConstantPropagate(ConstantPropagateOption.SHORTCUT).transform(parseContext)

      Hive-on-Tez does the same thing.

      Running the full constant propagation isn't really necessary, we just want to eliminate any and true predicates that were introduced by SyntheticJoinPredicate and DynamicPartitionPruningOptimization. The SyntheticJoinPredicate will introduce dummy filter predicates into the operator tree, and DynamicPartitionPruningOptimization will replace them. The predicates introduced via SyntheticJoinPredicate are necessary to help DynamicPartitionPruningOptimization determine if DPP can be used or not.

        Attachments

        1. HIVE-17405.7.patch
          240 kB
          Sahil Takiar
        2. HIVE-17405.6.patch
          240 kB
          Sahil Takiar
        3. HIVE-17405.5.patch
          241 kB
          Sahil Takiar
        4. HIVE-17405.4.patch
          184 kB
          Sahil Takiar
        5. HIVE-17405.3.patch
          135 kB
          Sahil Takiar
        6. HIVE-17405.2.patch
          4 kB
          Sahil Takiar
        7. HIVE-17405.1.patch
          1 kB
          Sahil Takiar

          Issue Links

            Activity

              People

              • Assignee:
                stakiar Sahil Takiar
                Reporter:
                stakiar Sahil Takiar
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: