Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25603

Generalize Nested Column Pruning

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • 3.0.0
    • None
    • SQL
    • None

    Attachments

      Issue Links

        links to

        Web Link GitHub Pull Request #23542

        1.
        		 Accessing nested fields with different cases in case insensitive mode Sub-task Resolved DB Tsai
        2.
        		 Refactor `ColumnPruning` from `Optimizer.scala` to `ColumnPruning.scala` Sub-task Resolved DB Tsai
        3.
        		 Prune the unused serializers from `SerializeFromObject` Sub-task Resolved L. C. Hsieh
        4.
        		 Pruning nested fields from object serializers Sub-task Resolved L. C. Hsieh
        5.
        		 Add NestedSchemaPruningBenchmark Sub-task Resolved Dongjoon Hyun
        6.
        		 Support nested-column pruning over limit/sample/repartition Sub-task Resolved Dongjoon Hyun
        7.
        		 Improve CollapseProject to handle projects cross limit/repartition/sample Sub-task Resolved Dongjoon Hyun
        8.
        		 Add ReadNestedSchemaTest for file-based data sources Sub-task Resolved Dongjoon Hyun
        9.
        		 Add AvroReadSchemaSuite Sub-task Resolved Dongjoon Hyun
        10.
        		 Add map_keys and map_values support in nested schema pruning Sub-task Resolved L. C. Hsieh
        11.
        		 Pruning nested serializers from object serializers: MapType support Sub-task Resolved L. C. Hsieh
        12.
        		 Pruning nested field in complex map key from object serializers Sub-task Resolved L. C. Hsieh
        13.
        		 Pruning nested field in map of map key and value from object serializers Sub-task Resolved L. C. Hsieh
        14.
        		 Update nested schema benchmark result for Orc V2 Sub-task Resolved L. C. Hsieh
        15.
        		 Nested schema pruning doesn't work for aggregation e.g. `sum`. Sub-task Resolved L. C. Hsieh
        16.
        		 catalyst inception of lateral view explode with struct raise a Catalyst error Sub-task Resolved Peter Toth
        17.
        		 PrquetRowConverter does not follow case sensitivity Sub-task Resolved Tae-kyeom, Kim
        18.
        		 Enable spark.sql.optimizer.nestedSchemaPruning.enabled by default Sub-task Resolved Unassigned
        19.
        		 Enable nested schema pruning and pruning on expressions by default Sub-task Resolved DB Tsai
        20.
        		 Nested column pruning for other operators Sub-task Resolved L. C. Hsieh

        Activity

          People

            dbtsai DB Tsai
            dbtsai DB Tsai
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated: