Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25603

Generalize Nested Column Pruning

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:
      None

      Attachments

        Issue Links

        1.
        Accessing nested fields with different cases in case insensitive mode Sub-task Resolved DB Tsai
        2.
        Refactor `ColumnPruning` from `Optimizer.scala` to `ColumnPruning.scala` Sub-task Resolved DB Tsai
        3.
        Prune the unused serializers from `SerializeFromObject` Sub-task Resolved L. C. Hsieh
        4.
        Pruning nested fields from object serializers Sub-task Resolved L. C. Hsieh
        5.
        Add NestedSchemaPruningBenchmark Sub-task Resolved Dongjoon Hyun
        6.
        Support nested-column pruning over limit/sample/repartition Sub-task Resolved Dongjoon Hyun
        7.
        Improve CollapseProject to handle projects cross limit/repartition/sample Sub-task Resolved Dongjoon Hyun
        8.
        Add ReadNestedSchemaTest for file-based data sources Sub-task Resolved Dongjoon Hyun
        9.
        Add AvroReadSchemaSuite Sub-task Resolved Dongjoon Hyun
        10.
        Add map_keys and map_values support in nested schema pruning Sub-task Resolved L. C. Hsieh
        11.
        Pruning nested serializers from object serializers: MapType support Sub-task Resolved L. C. Hsieh
        12.
        Pruning nested field in complex map key from object serializers Sub-task Resolved L. C. Hsieh
        13.
        Pruning nested field in map of map key and value from object serializers Sub-task Resolved L. C. Hsieh
        14.
        Update nested schema benchmark result for Orc V2 Sub-task Resolved L. C. Hsieh
        15.
        Nested schema pruning doesn't work for aggregation e.g. `sum`. Sub-task Resolved L. C. Hsieh
        16.
        catalyst inception of lateral view explode with struct raise a Catalyst error Sub-task Resolved Peter Toth
        17.
        PrquetRowConverter does not follow case sensitivity Sub-task Resolved Tae-kyeom, Kim
        18.
        Enable spark.sql.optimizer.nestedSchemaPruning.enabled by default Sub-task Resolved Unassigned
        19.
        Enable nested schema pruning and pruning on expressions by default Sub-task Resolved DB Tsai
        20.
        Nested column pruning for other operators Sub-task Resolved L. C. Hsieh

          Activity

            People

            • Assignee:
              dbtsai DB Tsai
              Reporter:
              dbtsai DB Tsai
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated: