Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-13540

Nested classes within Scala objects can't be used as Dataset element type

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0.0
    • 2.0.0
    • SQL
    • None

    Description

      Add the following test case to DatasetSuite to reproduce this issue:

      object OuterObject {
        case class InnerClass(a: String)
      }
      
      test("foo") {
        OuterScopes.addOuterScope(OuterObject)
        Seq(OuterObject.InnerClass("foo")).toDS()
      }
      

      Exception thrown:

      org.apache.spark.sql.AnalysisException: Unable to generate an encoder for inner class `org.apache.spark.sql.OuterObject$InnerClass` without access to the scope that this class was defined in.
      

      The reason is that an inner class within a Scala object is translated into a static nested class, and no outer scope is needed. But the analyzer still thinks there should be an outer scope.

      Attachments

        Activity

          People

            lian cheng Cheng Lian
            lian cheng Cheng Lian
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: