Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22386

Data Source V2 improvements

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.3.0
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:
    • Target Version/s:

      Attachments

        Issue Links

        depends upon

        Improvement - An improvement or enhancement to an existing feature or task. SPARK-25531 new write APIs for data source v2

        • Major - Major loss of function.
        • Open
        is related to

        Sub-task - The sub-task of the issue SPARK-23521 SPIP: Standardize SQL logical plans with DataSourceV2

        • Major - Major loss of function.
        • Resolved
        relates to

        Improvement - An improvement or enhancement to an existing feature or task. SPARK-26088 DataSourceV2 should expose row count and attribute statistics

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Open
        1.
        		 Limit push down Sub-task Open Unassigned
        2.
        		 Aggregate push down Sub-task Open Unassigned
        3.
        		 add `MetadataCreationSupport` trait to separate data and metadata handling at write path Sub-task Resolved Unassigned
        4.
        		 DataSourceV2 should use immutable trees. Sub-task Resolved Ryan Blue
        5.
        		 DataSourceV2 should support named tables in DataFrameReader, DataFrameWriter Sub-task Resolved Unassigned
        6.
        		 Reorganize packages in data source V2 Sub-task Resolved Gengliang Wang
        7.
        		 DataSourceV2 should apply some validation when writing. Sub-task Resolved Unassigned
        8.
        		 DataSourceV2 should use the output commit coordinator. Sub-task Resolved Ryan Blue
        9.
        		 DataSourceV2 readers should always produce InternalRow. Sub-task Resolved Ryan Blue
        10.
        		 DataSourceOptions should handle path and table names to avoid confusion. Sub-task Resolved Wenchen Fan
        11.
        		 use InternalRow in DataSourceWriter Sub-task Resolved Wenchen Fan
        12.
        		 DataSourceV2 should provide a way to get a source's schema. Sub-task Resolved Unassigned
        13.
        		 DataSourceV2 should not allow userSpecifiedSchema without ReadSupportWithSchema Sub-task Resolved Ryan Blue
        14.
        		 DataSourceV2: Rename DataReaderFactory to InputPartition. Sub-task Resolved Ryan Blue
        15.
        		 Data Source V2: Join Push Down Sub-task Open Unassigned
        16.
        		 DataSourceV2: Add catalog support Sub-task Resolved Ryan Blue
        17.
        		 DataSourceV2 should push filters and projection at physical plan conversion Sub-task Resolved Ryan Blue
        18.
        		 remove SupportsDeprecatedScanRow Sub-task Resolved Wenchen Fan
        19.
        		 Add support for USING syntax for DataSourceV2 Sub-task In Progress Unassigned
        20.
        		 merge ReadSupport and ReadSupportWithSchema Sub-task Resolved Wenchen Fan
        21.
        		 DataSourceV2: Remove SupportsPushDownCatalystFilters Sub-task Resolved Reynold Xin
        22.
        		 DataSourceV2: Add interfaces to pass required sorting and clustering for writes Sub-task Open Unassigned
        23.
        		 DataSourceV2: Structured Streaming does not respect SessionConfigSupport Sub-task Resolved Hyukjin Kwon
        24.
        		 Avoid to create a readsupport at write path in Data Source V2 Sub-task Resolved Hyukjin Kwon
        25.
        		 data source v2 API improvement Sub-task Resolved Wenchen Fan
        26.
        		 DataSourceV2: Support ALTER TABLE statements in catalyst SQL parser Sub-task Resolved Ryan Blue
        27.
        		 DataSourceV2: Add AlterTable v2 implementation Sub-task Closed Ryan Blue
        28.
        		 DataSourceV2: Support SHOW TABLES Sub-task Resolved Terry Kim
        29.
        		 Recover options and properties and pass them back into the v1 API Sub-task Open Unassigned
        30.
        		 DataSourceV2: Add new DataFrameWriter API for v2 Sub-task Resolved Ryan Blue
        31.
        		 DataSourceV2: Support SHOW DATABASES Sub-task Resolved Terry Kim
        32.
        		 Pass in number of partitions to BuildWriter Sub-task Resolved Ximo Guanter
        33.
        		 DataSourceV2: Support CREATE NAMESPACE Sub-task Resolved Terry Kim
        34.
        		 DataSourceV2: Support DROP NAMESPACE Sub-task Resolved Terry Kim
        35.
        		 DataSourceV2: Support SHOW CURRENT NAMESPACE Sub-task Resolved Terry Kim
        36.
        		 DataSourceV2: Add CURRENT_CATALOG function Sub-task Open Unassigned

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              cloud_fan Wenchen Fan
            • Votes:
              12 Vote for this issue
              Watchers:
              41 Start watching this issue

              Dates

              • Created:
                Updated: