Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25603 Generalize Nested Column Pruning
  3. SPARK-27217

Nested schema pruning doesn't work for aggregation e.g. `sum`.

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.0
    • 3.1.0
    • SQL
    • None

    Description

      Since SPARK-4502 is fixed, I would expect queries such as `select sum(b.x)` doesn't have to read other nested fields.

         
       rdd = spark.range(1000).rdd.map(lambda x: [x.id+3, [x.id+1, x.id-1]])
          df = spark.createDataFrame(add, schema='a:int,b:struct<x:int,y:int>')
          df.repartition(1).write.mode('overwrite').parquet('test.parquet')
          df = spark.read.parquet('test.parquet')
      
          spark.conf.set('spark.sql.optimizer.nestedSchemaPruning.enabled', 'true')
          df.select('b.x').explain()
          # ReadSchema: struct<b:struct<x:int>>
      
          spark.conf.set('spark.sql.optimizer.nestedSchemaPruning.enabled', 'false')
          df.select('b.x').explain()
          # ReadSchema: struct<b:struct<x:int,y:int>>
      
          spark.conf.set('spark.sql.optimizer.nestedSchemaPruning.enabled', 'true')
          df.selectExpr('sum(b.x)').explain()
          #  ReadSchema: struct<b:struct<x:int,y:int>>
      

      Attachments

        Activity

          aman_omer Aman Omer added a comment -

          I am able to reproduce this bug on spark-shell. Working on this

          aman_omer Aman Omer added a comment - I am able to reproduce this bug on spark-shell. Working on this
          apachespark Apache Spark added a comment -

          User 'viirya' has created a pull request for this issue:
          https://github.com/apache/spark/pull/28560

          apachespark Apache Spark added a comment - User 'viirya' has created a pull request for this issue: https://github.com/apache/spark/pull/28560
          apachespark Apache Spark added a comment -

          User 'viirya' has created a pull request for this issue:
          https://github.com/apache/spark/pull/28560

          apachespark Apache Spark added a comment - User 'viirya' has created a pull request for this issue: https://github.com/apache/spark/pull/28560
          gurwls223 Hyukjin Kwon added a comment - - edited

          Issue resolved by pull request 28560
          https://github.com/apache/spark/pull/28560

          gurwls223 Hyukjin Kwon added a comment - - edited Issue resolved by pull request 28560 https://github.com/apache/spark/pull/28560

          People

            viirya L. C. Hsieh
            colinfang colin fang
            Votes:
            4 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: