[SPARK-34210] Cannot create a record reader because of a previous error when spark accesses the hive on HBase table

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.0.1
• Component/s: SQL
• Assignee: Unassigned
• Reporter: zhangzhanchang
• Created: 23/Jan/21 03:50
• Updated: 01/Nov/22 11:05
• Resolved: 24/Jan/21 03:29

Description

It is normal for version 2.4.6 to use spark SQL to access hive on HBase table,Upgrade to spark3.0.1 with the following exception：

java.io.IOException: Cannot create a record reader because of a previous error. Please look at the previous logs lines from the task's full log for more details.
at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:252)
at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:131)
at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:276)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:272)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:276)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:272)

Caused by: java.lang.IllegalStateException: The input format instance has not been properly initialized. Ensure you call initializeTable either in your constructor or initialize method
at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getTable(TableInputFormatBase.java:585)
at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:247)
... 59 more
java.io.IOException: Cannot create a record reader because of a previous error. Please look at the previous logs lines from the task's full log for more details.

 

 

Issue Links

• duplicates: SPARK-32380 sparksql cannot access hive table while data in hbase
• is duplicated by: SPARK-36282 Spark ThriftServer does not search hive external table which storage in hbase
• links to: [Github] Pull Request #31302 (zzccctv)