Major Description
Since SPARK-4502 is fixed, I would expect queries such as `select sum(b.x)` doesn't have to read other nested fields.
rdd = spark.range(1000).rdd.map(lambda x: [x.id+3, [x.id+1, x.id-1]]) df = spark.createDataFrame(add, schema='a:int,b:struct<x:int,y:int>') df.repartition(1).write.mode('overwrite').parquet('test.parquet') df = spark.read.parquet('test.parquet') spark.conf.set('spark.sql.optimizer.nestedSchemaPruning.enabled', 'true') df.select('b.x').explain() # ReadSchema: struct<b:struct<x:int>> spark.conf.set('spark.sql.optimizer.nestedSchemaPruning.enabled', 'false') df.select('b.x').explain() # ReadSchema: struct<b:struct<x:int,y:int>> spark.conf.set('spark.sql.optimizer.nestedSchemaPruning.enabled', 'true') df.selectExpr('sum(b.x)').explain() # ReadSchema: struct<b:struct<x:int,y:int>>
