Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33134

Incorrect nested complex JSON fields raise an exception

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.2, 3.1.0
    • Fix Version/s: 3.0.2, 3.1.0
    • Component/s: SQL
    • Labels:
      None

      Description

      The code below:

      val pokerhand_raw = Seq("""[{"cards": [19], "playerId": 123456}]""").toDF("events")
          val event = new StructType()
            .add("playerId", LongType)
            .add("cards", ArrayType(
              new StructType()
                .add("id", LongType)
                .add("rank", StringType)))
          val pokerhand_events = pokerhand_raw
            .select(explode(from_json($"events", ArrayType(event))).as("event"))
          pokerhand_events.show
      

      throw the exception in the PERMISSIVE mode (default):

      Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to org.apache.spark.sql.catalyst.util.ArrayData
        at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow.getArray(rows.scala:48)
        at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow.getArray$(rows.scala:48)
        at org.apache.spark.sql.catalyst.expressions.GenericInternalRow.getArray(rows.scala:195)
        at org.apache.spark.sql.catalyst.expressions.JsonToStructs.$anonfun$converter$2(jsonExpressions.scala:560)
        at org.apache.spark.sql.catalyst.expressions.JsonToStructs.nullSafeEval(jsonExpressions.scala:597)
        at org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:461)
        at org.apache.spark.sql.catalyst.expressions.ExplodeBase.eval(generators.scala:313)
        at org.apache.spark.sql.execution.GenerateExec.$anonfun$doExecute$8(GenerateExec.scala:108)
      

      The same works in Spark 2.4:

      Welcome to
            ____              __
           / __/__  ___ _____/ /__
          _\ \/ _ \/ _ `/ __/  '_/
         /___/ .__/\_,_/_/ /_/\_\   version 2.4.6
            /_/
      
      Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_265)
      ...
      scala> pokerhand_events.show()
      +-----+
      |event|
      +-----+
      +-----+
      

        Attachments

          Activity

            People

            • Assignee:
              maxgekk Maxim Gekk
              Reporter:
              maxgekk Maxim Gekk
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: