Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26188

Spark 2.4.0 Partitioning behavior breaks backwards compatibility

    XMLWordPrintableJSON

Details

    Description

      My team uses spark to partition and output parquet files to amazon S3. We typically use 256 partitions, from 00 to ff.

      We've observed that in spark 2.3.2 and prior, it reads the partitions as strings by default. However, in spark 2.4.0 and later, the type of each partition is inferred by default, and partitions such as 00 become 0 and 4d become 4.0.
      Here is a log sample of this behavior from one of our jobs:
      2.4.0:

      18/11/27 14:02:27 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=00/part-00061-hashredacted.parquet, range: 0-662, partition values: [0]
18/11/27 14:02:28 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=ef/part-00034-hashredacted.parquet, range: 0-662, partition values: [ef]
18/11/27 14:02:29 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=4a/part-00151-hashredacted.parquet, range: 0-662, partition values: [4a]
18/11/27 14:02:30 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=74/part-00180-hashredacted.parquet, range: 0-662, partition values: [74]
18/11/27 14:02:32 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=f5/part-00156-hashredacted.parquet, range: 0-662, partition values: [f5]
18/11/27 14:02:33 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=50/part-00195-hashredacted.parquet, range: 0-662, partition values: [50]
18/11/27 14:02:34 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=70/part-00054-hashredacted.parquet, range: 0-662, partition values: [70]
18/11/27 14:02:35 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=b9/part-00012-hashredacted.parquet, range: 0-662, partition values: [b9]
18/11/27 14:02:37 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=d2/part-00016-hashredacted.parquet, range: 0-662, partition values: [d2]
18/11/27 14:02:38 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=51/part-00003-hashredacted.parquet, range: 0-662, partition values: [51]
18/11/27 14:02:39 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=84/part-00135-hashredacted.parquet, range: 0-662, partition values: [84]
18/11/27 14:02:40 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=b5/part-00190-hashredacted.parquet, range: 0-662, partition values: [b5]
18/11/27 14:02:41 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=88/part-00143-hashredacted.parquet, range: 0-662, partition values: [88]
18/11/27 14:02:42 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=4d/part-00120-hashredacted.parquet, range: 0-662, partition values: [4.0]
18/11/27 14:02:43 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=ac/part-00119-hashredacted.parquet, range: 0-662, partition values: [ac]
18/11/27 14:02:44 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=24/part-00139-hashredacted.parquet, range: 0-662, partition values: [24]
18/11/27 14:02:45 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=fd/part-00167-hashredacted.parquet, range: 0-662, partition values: [fd]
18/11/27 14:02:46 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=52/part-00033-hashredacted.parquet, range: 0-662, partition values: [52]
18/11/27 14:02:47 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=ab/part-00083-hashredacted.parquet, range: 0-662, partition values: [ab]
18/11/27 14:02:48 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=f8/part-00018-hashredacted.parquet, range: 0-662, partition values: [f8]
18/11/27 14:02:49 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=7a/part-00093-hashredacted.parquet, range: 0-662, partition values: [7a]
18/11/27 14:02:50 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=ba/part-00020-hashredacted.parquet, range: 0-662, partition values: [ba]
18/11/27 14:02:51 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=2d/part-00085-hashredacted.parquet, range: 0-662, partition values: [2.0]
18/11/27 14:02:52 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=03/part-00099-hashredacted.parquet, range: 0-662, partition values: [3]
18/11/27 14:02:53 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=57/part-00196-hashredacted.parquet, range: 0-662, partition values: [57]
18/11/27 14:02:54 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=81/part-00122-hashredacted.parquet, range: 0-662, partition values: [81]
18/11/27 14:02:55 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=6d/part-00167-hashredacted.parquet, range: 0-662, partition values: [6.0]
18/11/27 14:02:56 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=36/part-00154-hashredacted.parquet, range: 0-662, partition values: [36]
18/11/27 14:02:57 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=4b/part-00093-hashredacted.parquet, range: 0-662, partition values: [4b]

      2.3.2:

      18/11/27 14:09:00 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=60/part-00082-hashredacted.parquet, range: 0-662, partition values: [60]
18/11/27 14:09:01 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=00/part-00061-hashredacted.parquet, range: 0-662, partition values: [00]
18/11/27 14:09:02 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=ef/part-00034-hashredacted.parquet, range: 0-662, partition values: [ef]
18/11/27 14:09:02 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=4a/part-00151-hashredacted.parquet, range: 0-662, partition values: [4a]
18/11/27 14:09:03 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=74/part-00180-hashredacted.parquet, range: 0-662, partition values: [74]
18/11/27 14:09:04 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=f5/part-00156-hashredacted.parquet, range: 0-662, partition values: [f5]
18/11/27 14:09:04 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=50/part-00195-hashredacted.parquet, range: 0-662, partition values: [50]
18/11/27 14:09:05 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=70/part-00054-hashredacted.parquet, range: 0-662, partition values: [70]
18/11/27 14:09:05 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=b9/part-00012-hashredacted.parquet, range: 0-662, partition values: [b9]
18/11/27 14:09:06 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=d2/part-00016-hashredacted.parquet, range: 0-662, partition values: [d2]
18/11/27 14:09:06 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=51/part-00003-hashredacted.parquet, range: 0-662, partition values: [51]
18/11/27 14:09:07 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=84/part-00135-hashredacted.parquet, range: 0-662, partition values: [84]
18/11/27 14:09:08 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=b5/part-00190-hashredacted.parquet, range: 0-662, partition values: [b5]
18/11/27 14:09:08 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=88/part-00143-hashredacted.parquet, range: 0-662, partition values: [88]
18/11/27 14:09:09 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=4d/part-00120-hashredacted.parquet, range: 0-662, partition values: [4d]
18/11/27 14:09:09 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=ac/part-00119-hashredacted.parquet, range: 0-662, partition values: [ac]
18/11/27 14:09:10 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=24/part-00139-hashredacted.parquet, range: 0-662, partition values: [24]
18/11/27 14:09:11 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=fd/part-00167-hashredacted.parquet, range: 0-662, partition values: [fd]
18/11/27 14:09:11 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=52/part-00033-hashredacted.parquet, range: 0-662, partition values: [52]
18/11/27 14:09:12 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=ab/part-00083-hashredacted.parquet, range: 0-662, partition values: [ab]
18/11/27 14:09:12 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=f8/part-00018-hashredacted.parquet, range: 0-662, partition values: [f8]
18/11/27 14:09:13 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=7a/part-00093-hashredacted.parquet, range: 0-662, partition values: [7a]
18/11/27 14:09:13 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=ba/part-00020-hashredacted.parquet, range: 0-662, partition values: [ba]
18/11/27 14:09:14 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=2d/part-00085-hashredacted.parquet, range: 0-662, partition values: [2d]
18/11/27 14:09:15 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=03/part-00099-hashredacted.parquet, range: 0-662, partition values: [03]
18/11/27 14:09:15 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=57/part-00196-hashredacted.parquet, range: 0-662, partition values: [57]
18/11/27 14:09:16 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=81/part-00122-hashredacted.parquet, range: 0-662, partition values: [81]
18/11/27 14:09:17 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=6d/part-00167-hashredacted.parquet, range: 0-662, partition values: [6d]
18/11/27 14:09:17 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=36/part-00154-hashredacted.parquet, range: 0-662, partition values: [36]
18/11/27 14:09:18 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=4b/part-00093-hashredacted.parquet, range: 0-662, partition values: [4b]

      After some investigation, we've isolated the issue to
      https://github.com/apache/spark/blob/02b510728c31b70e6035ad541bfcdc2b59dcd79a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala#L132-L136
       

      In the inferPartitioning method, 2.3.2 sets the type inference to false by default:

      val spec = PartitioningUtils.parsePartitions(
  leafDirs,
  typeInference = false,
  basePaths = basePaths,
  timeZoneId = timeZoneId)

      However, in version 2.4.0, the typeInference flag has been replace with a config flag

      https://github.com/apache/spark/blob/075447b3965489ffba4e6afb2b120880bc307505/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala#L129-L133

       

      val inferredPartitionSpec = PartitioningUtils.parsePartitions(
  leafDirs,
  typeInference = sparkSession.sessionState.conf.partitionColumnTypeInferenceEnabled,
  basePaths = basePaths,
  timeZoneId = timeZoneId)

      And this conf's default value is true

      val PARTITION_COLUMN_TYPE_INFERENCE =
buildConf("spark.sql.sources.partitionColumnTypeInference.enabled")
.doc("When true, automatically infer the data types for partitioned columns.")
.booleanConf
.createWithDefault(true)

      https://github.com/apache/spark/blob/075447b3965489ffba4e6afb2b120880bc307505/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L636-L640
       

      I was wondering if a bug report would be appropriate to preserve backwards compatibility and change the default conf value to false.

       
       

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. SPARK-26990 Difference in handling of mixed-case partition column names after SPARK-26188

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #23165 [Github] Pull Request #23165 (gengliangwang)

          Web Link GitHub Pull Request #23165

          Activity

            People

              Gengliang.Wang Gengliang Wang
              ddgirard Damien Doucet-Girard
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: