Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25603 Generalize Nested Column Pruning
  3. SPARK-27217

Nested schema pruning doesn't work for aggregation e.g. `sum`.

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.1.0
    • Fix Version/s: 3.1.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Since SPARK-4502 is fixed, I would expect queries such as `select sum(b.x)` doesn't have to read other nested fields.

         
       rdd = spark.range(1000).rdd.map(lambda x: [x.id+3, [x.id+1, x.id-1]])
          df = spark.createDataFrame(add, schema='a:int,b:struct<x:int,y:int>')
          df.repartition(1).write.mode('overwrite').parquet('test.parquet')
          df = spark.read.parquet('test.parquet')
      
          spark.conf.set('spark.sql.optimizer.nestedSchemaPruning.enabled', 'true')
          df.select('b.x').explain()
          # ReadSchema: struct<b:struct<x:int>>
      
          spark.conf.set('spark.sql.optimizer.nestedSchemaPruning.enabled', 'false')
          df.select('b.x').explain()
          # ReadSchema: struct<b:struct<x:int,y:int>>
      
          spark.conf.set('spark.sql.optimizer.nestedSchemaPruning.enabled', 'true')
          df.selectExpr('sum(b.x)').explain()
          #  ReadSchema: struct<b:struct<x:int,y:int>>
      

        Attachments

          Activity

            People

            • Assignee:
              viirya L. C. Hsieh
              Reporter:
              colinfang colin fang

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment