[SPARK-31384] NPE in OptimizeSkewedJoin when there's a inputRDD of plan has 0 partition

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: wuyi
• Reporter: wuyi
• Created: 08/Apr/20 08:59
• Updated: 12/Apr/20 23:03
• Resolved: 10/Apr/20 08:36

Description

When there's a inputRDD of a plan with 0 partitions, rule OptimizeSkewedJoin can hit NPE.

The issue can be reproduced by below test:

withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true",
  SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {
  withTempView("t2") {
    // create DataFrame with 0 partition
    spark.createDataFrame(sparkContext.emptyRDD[Row], new StructType().add("b", IntegerType))
      .createOrReplaceTempView("t2")
    // should run successfully without NPE
    runAdaptiveAndVerifyResult("SELECT * FROM testData2 t1 left semi join t2 ON t1.a=t2.b")
  }
}

Issue Links

• links to: GitHub Pull Request #28153