Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39417

Handle Null partition values in PartitioningUtils

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.3.0, 3.4.0
    • SQL
    • None

    Description

      partitions with null values we get a NPE on partition discovery, earlier we use to get `DEFAULT_PARTITION_NAME`

       

      [info]   java.lang.NullPointerException:
      [info]   at org.apache.spark.sql.execution.datasources.PartitioningUtils$.removeLeadingZerosFromNumberTypePartition(PartitioningUtils.scala:362)
      [info]   at org.apache.spark.sql.execution.datasources.PartitioningUtils$.$anonfun$getPathFragment$1(PartitioningUtils.scala:355)
      [info]   at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
      [info]   at scala.collection.Iterator.foreach(Iterator.scala:943)
      [info]   at scala.collection.Iterator.foreach$(Iterator.scala:943)

      Attachments

        Activity

          People

            singhpk234 Prashant Singh
            singhpk234 Prashant Singh
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: