Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26845

Avro to_avro from_avro roundtrip fails if data type is string

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Not A Problem
    • Affects Version/s: 2.4.0, 3.0.0
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:

      Description

      I was playing with AvroFunctionsSuite and created a situation where test fails which I believe it shouldn't:

        test("roundtrip in to_avro and from_avro - string") {
          val df = spark.createDataset(Seq("1", "2", "3")).select('value.cast("string").as("str"))
      
          val avroDF = df.select(to_avro('str).as("b"))
          val avroTypeStr = s"""
            |{
            |  "type": "string",
            |  "name": "str"
            |}
          """.stripMargin
          checkAnswer(avroDF.select(from_avro('b, avroTypeStr)), df)
        }
      
      == Results ==
      !== Correct Answer - 3 ==   == Spark Answer - 3 ==
      !struct<str:string>         struct<from_avro(b):string>
      ![1]                        []
      ![2]                        []
      ![3]                        []
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              gsomogyi Gabor Somogyi
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: