[SPARK-47489] Action performed on Dataset is triggering an error as "The Spark SQL phase planning failed with an internal error."

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.2
• Component/s: k8s, Spark Core, Spark Submit
• Assignee: Unassigned
• Reporter: Jagadeesh Marada
• Created: 20/Mar/24 16:47
• Updated: 27/Mar/24 10:39

Description

Hi team, 

We are upgrading to spark 3.4.2 from 3.3.2 and facing issue in the spark driver as below.

"org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase planning failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace."

 

Complete stack trace is as attached.

After analysing further , found that when we try to perform any actions on the dataset, spark is unable to plan its actions, results in this exception. 

Attaching a Exception stack trace & code snippet where exactly we are trigging an action to convert it as java RDD.

 

Attaching the spark submit command as well for your reference.

Can you please check this issue let us know if any fix available for this issue.