Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19982

JavaDatasetSuite.testJavaBeanEncoder sometimes fails with "Unable to generate an encoder for inner class"

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Incomplete
    • Affects Version/s: 2.1.0
    • Fix Version/s: None
    • Component/s: Tests

      Description

      JavaDatasetSuite.testJavaBeanEncoder fails sporadically with the error below:

      Unable to generate an encoder for inner class `test.org.apache.spark.sql.JavaDatasetSuite$SimpleJavaBean` without access to the scope that this class was defined in. Try moving this class out of its parent class.

      From https://spark-tests.appspot.com/test-logs/35475788

      Marcelo Masiero Vanzin looked into this back in October and reported:

      I ran this test in a loop (both alone and with the rest of the spark-sql tests) and never got a failure. I even used the same JDK as Jenkins (1.7.0_51).
      Also looked at the code and nothing seems wrong. The errors is when an entry with the parent class name is missing from the map kept in OuterScopes.scala, but the test populates that map in its first line. So it doesn't look like a race nor some issue with weak references (the map uses weak values).

      public void testJavaBeanEncoder() {
      OuterScopes.addOuterScope(this);

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jsoltren Jose Soltren
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: