[HIVE-8233] multi-table insertion doesn't work with ForwardOperator [Spark Branch]

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: Spark
• Assignee: Chao Sun
• Reporter: Chao Sun
• Created: 23/Sep/14 17:45
• Updated: 16/Oct/14 23:10
• Resolved: 16/Oct/14 23:10

Description

Right now, for multi-table insertion, we will start from multiple FileSinkOperators, and break from their lowest common ancestor, adding temporary FileSinkOperator and TableScanOperators. A special case is when the LCA is a ForwardOperator, in which case we don't break it, since it's already been optimized.

However, there's a issue, considering the following plan:

      ...
      RS_0
       |
      FOR
       |
     /   \
   GBY_1  GBY_2
    |     |
   ...   ...
    |     |
   RS_1  RS_2
    |     |
   ...   ...
    |     |
   FS_1  FS_2

which may result to:

          RW
         /  \
       RW    RW

Hence, because of the issue in HIVE-7731 and HIVE-8118, both downstream branches will get duplicated (and same) input.

Issue Links

• depends upon: HIVE-7503 Support Hive's multi-table insert query with Spark [Spark Branch]
• Is contained by: HIVE-7292 Hive on Spark
• is related to: HIVE-8207 Add .q tests for multi-table insertion [Spark Branch]
• relates to: HIVE-8242 Investigate test failures when hive.multigroupby.singlereducer and hive.optimize.multigroupby.common.distincts are set to false [Spark Branch]
• links to: RB Link