[HIVE-3326] plan for multiple mapjoin followed by a normal join is wrong

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Component/s: SQL

Environment

OS X 10.8; java 1.6.0_33

• Assignee: Navis Ryu
• Reporter: Zhang Xinyu
• Created: 01/Aug/12 06:52
• Updated: 25/Feb/13 06:52
• Resolved: 25/Feb/13 06:52

Description

example queries:

create table yudi(c1 int, c2 int, c3 int, c4 int);
create table wangmu(c1 int, c2 int, c3 int, c4 int);
select /*+mapjoin(b,c)*/ * from yudi a join yudi b on a.c1=b.c1 join wangmu c on b.c2=c.c2 join yudi d on a.c3=d.c3;

in explain mode, I got this:

hive> explain select /*+mapjoin(b,c)*/ * from yudi a join yudi b on a.c1=b.c1 join wangmu c on b.c2=c.c2 join yudi d on a.c3=d.c3;
OK
STAGE DEPENDENCIES:
  Stage-8 is a root stage
  Stage-2 depends on stages: Stage-8
  Stage-7 depends on stages: Stage-2
  Stage-3 depends on stages: Stage-7
  Stage-1 depends on stages: Stage-3

STAGE PLANS:
  Stage: Stage-8
    Map Reduce Local Work
      Alias -> Map Local Tables:
        b
        <Not Important>
  Stage: Stage-2
    Map Reduce
      Alias -> Map Operator Tree:
        a
        <Not Important>
      Local Work:
        Map Reduce Local Work

  Stage: Stage-7
    Map Reduce Local Work
      Alias -> Map Local Tables:
        c
        <Not Important>
  Stage: Stage-3
    Map Reduce
      Alias -> Map Operator Tree:
           file:/var/folders/4w/3_nk1cwd4pd023mzx64p3r480000gn/T/dukezhang/hive_2012-08-01_14-01-37_152_5814747325029961632/-mr-10002
        <Not Important>
      Local Work:
        Map Reduce Local Work

  Stage: Stage-1
    Map Reduce
      Alias -> Map Operator Tree:
        d
          TableScan

        file:/var/folders/4w/3_nk1cwd4pd023mzx64p3r480000gn/T/dukezhang/hive_2012-08-01_14-01-37_152_5814747325029961632/-mr-10002
          Select Operator

      Reduce Operator Tree:
      <Not Important>

You see, mapper of Stage-1 should read from Stage-3, maybe '.../-mr-10003', not Stage-2(result in '.../-mr-10002').

To resolve this bug, I found these codes(GenMapRedUtils.java, about line 431):

GenMapRedUtils.java

if (oldMapJoin == null) {
  if (opProcCtx.getParseCtx().getListMapJoinOpsNoReducer().contains(mjOp)
      || local || (oldTask != null) && (parTasks != null)) {
    taskTmpDir = mjCtx.getTaskTmpDir();
    tt_desc = mjCtx.getTTDesc();
    rootOp = mjCtx.getRootMapJoinOp();
    }
} else {
  GenMRMapJoinCtx oldMjCtx = opProcCtx.getMapJoinCtx(oldMapJoin);
  assert oldMjCtx != null;
  taskTmpDir = oldMjCtx.getTaskTmpDir();
  tt_desc = oldMjCtx.getTTDesc();
  rootOp = oldMjCtx.getRootMapJoinOp();
}

my query goes into 'else' block and gets wrong taskTmpDir. I hack them to let query go into 'if' block, and it works.

Issue Links

• is related to: HIVE-3784 de-emphasize mapjoin hint