[SPARK-26630] Support reading Hive-serde tables whose INPUTFORMAT is org.apache.hadoop.mapreduce

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0, 2.4.1, 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: Deegue
• Reporter: Deegue
• Created: 16/Jan/19 04:46
• Updated: 27/Jan/19 02:52
• Resolved: 26/Jan/19 18:20

Description

This bug found in link title (PR #23506).

It will throw ClassCastException when we use new input format (eg. `org.apache.hadoop.mapreduce.InputFormat`) to create HadoopRDD.So we need to use NewHadoopRDD to deal with this input format in TableReader.scala.

Exception :

Caused by: java.lang.ClassCastException: org.apache.hadoop.mapreduce.lib.input.TextInputFormat cannot be cast to org.apache.hadoop.mapred.InputFormat
	at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:190)
	at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:204)
	at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:254)
	at scala.Option.getOrElse(Option.scala:138)
	at org.apache.spark.rdd.RDD.partitions(RDD.scala:252)
	at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
	at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:254)
	at scala.Option.getOrElse(Option.scala:138)
	at org.apache.spark.rdd.RDD.partitions(RDD.scala:252)
	at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
	at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:254)
	at scala.Option.getOrElse(Option.scala:138)
	at org.apache.spark.rdd.RDD.partitions(RDD.scala:252)
	at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
	at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:254)
	at scala.Option.getOrElse(Option.scala:138)
	at org.apache.spark.rdd.RDD.partitions(RDD.scala:252)
	at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
	at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:254)
	at scala.Option.getOrElse(Option.scala:138)
	at org.apache.spark.rdd.RDD.partitions(RDD.scala:252)
	at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
	at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:254)
	at scala.Option.getOrElse(Option.scala:138)
	at org.apache.spark.rdd.RDD.partitions(RDD.scala:252)
	at org.apache.spark.ShuffleDependency.<init>(Dependency.scala:96)
	at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$.prepareShuffleDependency(ShuffleExchangeExec.scala:343)
	at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.prepareShuffleDependency(ShuffleExchangeExec.scala:101)
	at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.$anonfun$doExecute$1(ShuffleExchangeExec.scala:137)
	at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:52)
	... 87 more

Issue Links

• links to: GitHub Pull Request #23559