[HIVE-22294] ConditionalWork cannot be cast to MapredWork When both skew.join and auto.convert is on.

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 2.3.0, 2.3.4, 3.1.1
• Component/s: Physical Optimizer
• Labels: pull-request-available
• Assignee: Nemon Lou
• Reporter: Qiang.Kang
• Created: 05/Oct/19 05:27
• Updated: 01/Dec/21 00:17

Description

Our hive version is 1.2.1 which has merged some patches (including patches mentioned  in https://issues.apache.org/jira/browse/HIVE-14557, https://issues.apache.org/jira/browse/HIVE-16155 ) .

 

My sql query string is like this:

// code placeholder
set hive.auto.convert.join = true;
set hive.optimize.skewjoin=true;
 
SELECT a.*
FROM
a
JOIN b
ON a.id=b.id AND a.uid = b.uid 
LEFT JOIN c
ON b.id=c.id AND b.uid=c.uid;
 

 

And we met some error: 

FAILED: ClassCastException org.apache.hadoop.hive.ql.plan.ConditionalWork cannot be cast to org.apache.hadoop.hive.ql.plan.MapredWork

 

The main reason is that there is a conditional task (MapJoin) in the list tasks of another Conditional task (SkewJoin).  Here is the code snippet where it throws this exception:

`org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver:`

 

// code placeholder
public Object dispatch(Node nd, Stack<Node> stack, Object... nodeOutputs)
 throws SemanticException {
 Task<? extends Serializable> currTask = (Task<? extends Serializable>) nd;
 // not map reduce task or not conditional task, just skip
 if (currTask.isMapRedTask()) {
 if (currTask instanceof ConditionalTask) {
 // get the list of task
 List<Task<? extends Serializable>> taskList = ((ConditionalTask) currTask).getListTasks();
 for (Task<? extends Serializable> tsk : taskList) {
 if (tsk.isMapRedTask())
{   //  ATTENTION: tsk May be ConditionalTask !!! this.processCurrentTask(tsk, ((ConditionalTask) currTask)); }
}
 } else
{ this.processCurrentTask(currTask, null); }
}
 return null;
 }
private void processCurrentTask(Task<? extends Serializable> currTask,
 ConditionalTask conditionalTask) throws SemanticException {
 // get current mapred work and its local work
 MapredWork mapredWork = (MapredWork) currTask.getWork(); // WRONG!!!!!!
 MapredLocalWork localwork = mapredWork.getMapWork().getMapRedLocalWork();
 

 

Here is some detail Information about query plan:

• •  set hive.auto.convert.join = true; set hive.optimize.skewjoin=false;*

// code placeholder

Stage-1 is a root stage [a join b]
 Stage-12 [map join]depends on stages: Stage-1 , consists of Stage-13, Stage-2
 Stage-13 has a backup stage: Stage-2
 Stage-11 depends on stages: Stage-13
 Stage-8 depends on stages: Stage-2, Stage-11 , consists of Stage-5, Stage-4, Stage-6
 Stage-5
 Stage-0 depends on stages: Stage-5, Stage-4, Stage-7
 Stage-14 depends on stages: Stage-0
 Stage-3 depends on stages: Stage-14
 Stage-4
 Stage-6
 Stage-7 depends on stages: Stage-6
 Stage-2
 

• •  set hive.auto.convert.join = false; set hive.optimize.skewjoin=true;*

// code placeholder

STAGE DEPENDENCIES:
 Stage-1 is a root stage
 Stage-12 depends on stages: Stage-1 , consists of Stage-13, Stage-2
 Stage-13 [skew Join map local task]
 Stage-11 depends on stages: Stage-13
 Stage-2 depends on stages: Stage-11
 Stage-8 depends on stages: Stage-2 , consists of Stage-5, Stage-4, Stage-6
 Stage-5
 Stage-0 depends on stages: Stage-5, Stage-4, Stage-7
 Stage-14 depends on stages: Stage-0
 Stage-3 depends on stages: Stage-14
 Stage-4
 Stage-6
 Stage-7 depends on stages: Stage-6

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Issue Links

• links to: GitHub Pull Request #2673