Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-48248

Fix nested array to respect legacy conf of inferArrayTypeFromFirstElement

    XMLWordPrintableJSON

Details

    Description

      >>> spark.conf.set("spark.sql.pyspark.legacy.inferArrayTypeFromFirstElement.enabled", True)
      >>> spark.createDataFrame([[[[1, "a"]]]])
      DataFrame[_1: array<array<string>>]
      

      should infer it as an integer of array

      Attachments

        Activity

          People

            gurwls223 Hyukjin Kwon
            gurwls223 Hyukjin Kwon
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: