[SPARK-46373] Create DataFrame Bug

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.5.0
• Component/s: Spark Core

Language

scala

• Assignee: Unassigned
• Reporter: Bleibtreu
• Created: 12/Dec/23 03:16
• Updated: 25/Jan/24 19:52

Description

Scala version is 3.3.1

Spark version is 3.5.0

I am using spark-core 3.5.1. I am trying to create a DataFrame through the reflection api, but "No TypeTag available for Person" will appear. I have tried for a long time, but I still don't quite understand why TypeTag cannot recognize my Person case class. 

    import sparkSession.implicits._
    import scala.reflect.runtime.universe._
    case class Person(name: String)
    val a = List(Person("A"), Person("B"), Person("C"))
    val df = sparkSession.createDataFrame(a)
    df.show()

I tested it and it is indeed a problem unique to Scala3
There is no problem on Scala2.13