Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-8208

Multi-table insertion optimization #1: don't always break operator tree. [Spark Branch]

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Currently, with the current patch of multi-table insertion, it will break whenever there exists one TableScanOperator that can leads to multiple FileSinkOperators. Then, it identifies the lowest common ancestor (LCA), and breaks the tree there, creating same number of child SparkTasks as the number of FileSinkOperators.

      However, in the following situation it's better not to break the operator tree:

      Of all the paths from these FileSinkOperators to the LCA, if ReduceSinkOperator only exist in 0 or 1 path of them.

      In this case, we can do it in one spark job, and no need to break the operator tree.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                csun Chao Sun
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: