[SPARK-36947] Exception when trying to access Row field using getAs method

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.2
• Component/s: SQL
• Labels: catalyst row sql

Environment

Spark 3.1.2 (but this also may affect other versions as well)

Language

scala

• Assignee: Unassigned
• Reporter: Alexandros Mavrommatis
• Created: 07/Oct/21 15:03
• Updated: 08/Oct/21 00:26

Description

I have an input dataframe df with the following schema:

|-- origin: string (nullable = true)
|-- product: struct (nullable = true)
|    |-- id: integer (nullable = true)

 

when I try to select the first 20 rows of the id column I execute:

df.select("product.id").show(20, false)

 

and I manage to get the result. But when I execute the following: 

df.map(_.getAs[Int]("product.id")).show(20, false)

 

I get the following error:

java.lang.IllegalArgumentException: Field "product.id" does not exist.