[SPARK-25603] Generalize Nested Column Pruning

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: SQL
• Assignee: DB Tsai
• Reporter: DB Tsai
• Created: 03/Oct/18 14:10
• Updated: 24/Feb/20 22:50

Issue Links

• links to: GitHub Pull Request #23542

Sub-Tasks

• Accessing nested fields with different cases in case insensitive mode: Resolved
• Refactor `ColumnPruning` from `Optimizer.scala` to `ColumnPruning.scala`: Resolved
• Prune the unused serializers from `SerializeFromObject`: Resolved
• Pruning nested fields from object serializers: Resolved
• Add NestedSchemaPruningBenchmark: Resolved
• Support nested-column pruning over limit/sample/repartition: Resolved
• Improve CollapseProject to handle projects cross limit/repartition/sample: Resolved
• Add ReadNestedSchemaTest for file-based data sources: Resolved
• Add AvroReadSchemaSuite: Resolved
• Add map_keys and map_values support in nested schema pruning: Resolved
• Pruning nested serializers from object serializers: MapType support: Resolved
• Pruning nested field in complex map key from object serializers: Resolved
• Pruning nested field in map of map key and value from object serializers: Resolved
• Update nested schema benchmark result for Orc V2: Resolved
• Nested schema pruning doesn't work for aggregation e.g. `sum`.: Resolved
• catalyst inception of lateral view explode with struct raise a Catalyst error: Resolved
• PrquetRowConverter does not follow case sensitivity: Resolved
• Enable spark.sql.optimizer.nestedSchemaPruning.enabled by default: Resolved
• Enable nested schema pruning and pruning on expressions by default: Resolved
• Nested column pruning for other operators: Resolved