[SPARK-12624] When schema is specified, we should give better error message if actual row length doesn't match

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.6.1, 2.0.0
• Component/s: PySpark, SQL
• Target Version/s: 1.6.1, 2.0.0
• Assignee: Cheng Lian
• Reporter: Reynold Xin
• Created: 04/Jan/16 19:50
• Updated: 19/Jun/16 07:23
• Resolved: 25/Jan/16 03:41

Description

The following code snippet reproduces this issue:

from pyspark.sql.types import StructType, StructField, IntegerType, StringType
from pyspark.sql.types import Row

schema = StructType([StructField("a", IntegerType()), StructField("b", StringType())])
rdd = sc.parallelize(range(10)).map(lambda x: Row(a=x))
df = sqlContext.createDataFrame(rdd, schema)
df.show()

An unintuitive ArrayIndexOutOfBoundsException exception is thrown in this case:

...
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
        at org.apache.spark.sql.catalyst.expressions.GenericInternalRow.genericGet(rows.scala:227)
        at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow$class.getAs(rows.scala:35)
        at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow$class.isNullAt(rows.scala:36)
...

We should give a better error message here.

Issue Links

• relates to: SPARK-13748 Document behavior of createDataFrame and rows with omitted fields
• links to: [Github] Pull Request #10886 (liancheng)