Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26188

Spark 2.4.0 Partitioning behavior breaks backwards compatibility

    XMLWordPrintableJSON

    Details

    • Target Version/s:

      Description

      My team uses spark to partition and output parquet files to amazon S3. We typically use 256 partitions, from 00 to ff.

      We've observed that in spark 2.3.2 and prior, it reads the partitions as strings by default. However, in spark 2.4.0 and later, the type of each partition is inferred by default, and partitions such as 00 become 0 and 4d become 4.0.
      Here is a log sample of this behavior from one of our jobs:
      2.4.0:

      18/11/27 14:02:27 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=00/part-00061-hashredacted.parquet, range: 0-662, partition values: [0]
      18/11/27 14:02:28 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=ef/part-00034-hashredacted.parquet, range: 0-662, partition values: [ef]
      18/11/27 14:02:29 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=4a/part-00151-hashredacted.parquet, range: 0-662, partition values: [4a]
      18/11/27 14:02:30 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=74/part-00180-hashredacted.parquet, range: 0-662, partition values: [74]
      18/11/27 14:02:32 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=f5/part-00156-hashredacted.parquet, range: 0-662, partition values: [f5]
      18/11/27 14:02:33 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=50/part-00195-hashredacted.parquet, range: 0-662, partition values: [50]
      18/11/27 14:02:34 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=70/part-00054-hashredacted.parquet, range: 0-662, partition values: [70]
      18/11/27 14:02:35 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=b9/part-00012-hashredacted.parquet, range: 0-662, partition values: [b9]
      18/11/27 14:02:37 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=d2/part-00016-hashredacted.parquet, range: 0-662, partition values: [d2]
      18/11/27 14:02:38 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=51/part-00003-hashredacted.parquet, range: 0-662, partition values: [51]
      18/11/27 14:02:39 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=84/part-00135-hashredacted.parquet, range: 0-662, partition values: [84]
      18/11/27 14:02:40 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=b5/part-00190-hashredacted.parquet, range: 0-662, partition values: [b5]
      18/11/27 14:02:41 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=88/part-00143-hashredacted.parquet, range: 0-662, partition values: [88]
      18/11/27 14:02:42 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=4d/part-00120-hashredacted.parquet, range: 0-662, partition values: [4.0]
      18/11/27 14:02:43 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=ac/part-00119-hashredacted.parquet, range: 0-662, partition values: [ac]
      18/11/27 14:02:44 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=24/part-00139-hashredacted.parquet, range: 0-662, partition values: [24]
      18/11/27 14:02:45 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=fd/part-00167-hashredacted.parquet, range: 0-662, partition values: [fd]
      18/11/27 14:02:46 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=52/part-00033-hashredacted.parquet, range: 0-662, partition values: [52]
      18/11/27 14:02:47 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=ab/part-00083-hashredacted.parquet, range: 0-662, partition values: [ab]
      18/11/27 14:02:48 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=f8/part-00018-hashredacted.parquet, range: 0-662, partition values: [f8]
      18/11/27 14:02:49 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=7a/part-00093-hashredacted.parquet, range: 0-662, partition values: [7a]
      18/11/27 14:02:50 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=ba/part-00020-hashredacted.parquet, range: 0-662, partition values: [ba]
      18/11/27 14:02:51 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=2d/part-00085-hashredacted.parquet, range: 0-662, partition values: [2.0]
      18/11/27 14:02:52 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=03/part-00099-hashredacted.parquet, range: 0-662, partition values: [3]
      18/11/27 14:02:53 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=57/part-00196-hashredacted.parquet, range: 0-662, partition values: [57]
      18/11/27 14:02:54 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=81/part-00122-hashredacted.parquet, range: 0-662, partition values: [81]
      18/11/27 14:02:55 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=6d/part-00167-hashredacted.parquet, range: 0-662, partition values: [6.0]
      18/11/27 14:02:56 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=36/part-00154-hashredacted.parquet, range: 0-662, partition values: [36]
      18/11/27 14:02:57 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=4b/part-00093-hashredacted.parquet, range: 0-662, partition values: [4b]

      2.3.2:

      18/11/27 14:09:00 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=60/part-00082-hashredacted.parquet, range: 0-662, partition values: [60]
      18/11/27 14:09:01 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=00/part-00061-hashredacted.parquet, range: 0-662, partition values: [00]
      18/11/27 14:09:02 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=ef/part-00034-hashredacted.parquet, range: 0-662, partition values: [ef]
      18/11/27 14:09:02 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=4a/part-00151-hashredacted.parquet, range: 0-662, partition values: [4a]
      18/11/27 14:09:03 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=74/part-00180-hashredacted.parquet, range: 0-662, partition values: [74]
      18/11/27 14:09:04 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=f5/part-00156-hashredacted.parquet, range: 0-662, partition values: [f5]
      18/11/27 14:09:04 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=50/part-00195-hashredacted.parquet, range: 0-662, partition values: [50]
      18/11/27 14:09:05 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=70/part-00054-hashredacted.parquet, range: 0-662, partition values: [70]
      18/11/27 14:09:05 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=b9/part-00012-hashredacted.parquet, range: 0-662, partition values: [b9]
      18/11/27 14:09:06 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=d2/part-00016-hashredacted.parquet, range: 0-662, partition values: [d2]
      18/11/27 14:09:06 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=51/part-00003-hashredacted.parquet, range: 0-662, partition values: [51]
      18/11/27 14:09:07 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=84/part-00135-hashredacted.parquet, range: 0-662, partition values: [84]
      18/11/27 14:09:08 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=b5/part-00190-hashredacted.parquet, range: 0-662, partition values: [b5]
      18/11/27 14:09:08 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=88/part-00143-hashredacted.parquet, range: 0-662, partition values: [88]
      18/11/27 14:09:09 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=4d/part-00120-hashredacted.parquet, range: 0-662, partition values: [4d]
      18/11/27 14:09:09 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=ac/part-00119-hashredacted.parquet, range: 0-662, partition values: [ac]
      18/11/27 14:09:10 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=24/part-00139-hashredacted.parquet, range: 0-662, partition values: [24]
      18/11/27 14:09:11 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=fd/part-00167-hashredacted.parquet, range: 0-662, partition values: [fd]
      18/11/27 14:09:11 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=52/part-00033-hashredacted.parquet, range: 0-662, partition values: [52]
      18/11/27 14:09:12 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=ab/part-00083-hashredacted.parquet, range: 0-662, partition values: [ab]
      18/11/27 14:09:12 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=f8/part-00018-hashredacted.parquet, range: 0-662, partition values: [f8]
      18/11/27 14:09:13 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=7a/part-00093-hashredacted.parquet, range: 0-662, partition values: [7a]
      18/11/27 14:09:13 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=ba/part-00020-hashredacted.parquet, range: 0-662, partition values: [ba]
      18/11/27 14:09:14 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=2d/part-00085-hashredacted.parquet, range: 0-662, partition values: [2d]
      18/11/27 14:09:15 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=03/part-00099-hashredacted.parquet, range: 0-662, partition values: [03]
      18/11/27 14:09:15 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=57/part-00196-hashredacted.parquet, range: 0-662, partition values: [57]
      18/11/27 14:09:16 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=81/part-00122-hashredacted.parquet, range: 0-662, partition values: [81]
      18/11/27 14:09:17 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=6d/part-00167-hashredacted.parquet, range: 0-662, partition values: [6d]
      18/11/27 14:09:17 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=36/part-00154-hashredacted.parquet, range: 0-662, partition values: [36]
      18/11/27 14:09:18 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=4b/part-00093-hashredacted.parquet, range: 0-662, partition values: [4b]
      
      

      After some investigation, we've isolated the issue to
      https://github.com/apache/spark/blob/02b510728c31b70e6035ad541bfcdc2b59dcd79a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala#L132-L136
       

      In the inferPartitioning method, 2.3.2 sets the type inference to false by default:

      val spec = PartitioningUtils.parsePartitions(
        leafDirs,
        typeInference = false,
        basePaths = basePaths,
        timeZoneId = timeZoneId)

      However, in version 2.4.0, the typeInference flag has been replace with a config flag

      https://github.com/apache/spark/blob/075447b3965489ffba4e6afb2b120880bc307505/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala#L129-L133

       

      val inferredPartitionSpec = PartitioningUtils.parsePartitions(
        leafDirs,
        typeInference = sparkSession.sessionState.conf.partitionColumnTypeInferenceEnabled,
        basePaths = basePaths,
        timeZoneId = timeZoneId)

      And this conf's default value is true

      val PARTITION_COLUMN_TYPE_INFERENCE =
      buildConf("spark.sql.sources.partitionColumnTypeInference.enabled")
      .doc("When true, automatically infer the data types for partitioned columns.")
      .booleanConf
      .createWithDefault(true)

      https://github.com/apache/spark/blob/075447b3965489ffba4e6afb2b120880bc307505/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L636-L640
       

      I was wondering if a bug report would be appropriate to preserve backwards compatibility and change the default conf value to false.

       
       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Gengliang.Wang Gengliang Wang
                Reporter:
                ddgirard Damien Doucet-Girard
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: