Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22386

Data Source V2 improvements

          Details

          • Type: Improvement
          • Status: In Progress
          • Priority: Major
          • Resolution: Unresolved
          • Affects Version/s: 2.3.0
          • Fix Version/s: None
          • Component/s: SQL
          • Labels:
          • Target Version/s:

            Attachments

              Issue Links

              depends upon

              Improvement - An improvement or enhancement to an existing feature or task. SPARK-25531 new write APIs for data source v2

              • Major - Major loss of function.
              • Open
              is related to

              Sub-task - The sub-task of the issue SPARK-23521 SPIP: Standardize SQL logical plans with DataSourceV2

              • Major - Major loss of function.
              • Resolved
              relates to

              Improvement - An improvement or enhancement to an existing feature or task. SPARK-26088 DataSourceV2 should expose row count and attribute statistics

              • Minor - Minor loss of function, or other problem where easy workaround is present.
              • Open
              1.
              		 Limit push down Sub-task Open Unassigned
              2.
              		 Aggregate push down Sub-task Open Unassigned
              3.
              		 add `MetadataCreationSupport` trait to separate data and metadata handling at write path Sub-task Open Unassigned
              4.
              		 DataSourceV2 should use immutable trees. Sub-task Resolved Ryan Blue
              5.
              		 DataSourceV2 should support named tables in DataFrameReader, DataFrameWriter Sub-task In Progress Unassigned
              6.
              		 Reorganize packages in data source V2 Sub-task Resolved Gengliang Wang
              7.
              		 DataSourceV2 should apply some validation when writing. Sub-task In Progress Unassigned
              8.
              		 DataSourceV2 should use the output commit coordinator. Sub-task Resolved Ryan Blue
              9.
              		 DataSourceV2 readers should always produce InternalRow. Sub-task Resolved Ryan Blue
              10.
              		 DataSourceOptions should handle path and table names to avoid confusion. Sub-task Resolved Wenchen Fan
              11.
              		 use InternalRow in DataSourceWriter Sub-task Resolved Wenchen Fan
              12.
              		 DataSourceV2 should provide a way to get a source's schema. Sub-task Open Unassigned
              13.
              		 DataSourceV2 should not allow userSpecifiedSchema without ReadSupportWithSchema Sub-task Resolved Ryan Blue
              14.
              		 DataSourceV2: Rename DataReaderFactory to InputPartition. Sub-task Resolved Ryan Blue
              15.
              		 Data Source V2: Join Push Down Sub-task Open Unassigned
              16.
              		 DataSourceV2: Add catalog support Sub-task In Progress Unassigned
              17.
              		 DataSourceV2 should push filters and projection at physical plan conversion Sub-task Resolved Ryan Blue
              18.
              		 remove SupportsDeprecatedScanRow Sub-task Resolved Wenchen Fan
              19.
              		 merge ReadSupport and ReadSupportWithSchema Sub-task Resolved Wenchen Fan
              20.
              		 DataSourceV2: Remove SupportsPushDownCatalystFilters Sub-task Resolved Reynold Xin
              21.
              		 DataSourceV2: Add interfaces to pass required sorting and clustering for writes Sub-task Open Unassigned
              22.
              		 Add support for USING syntax for DataSourceV2 Sub-task In Progress Unassigned
              23.
              		 DataSourceV2: Structured Streaming does not respect SessionConfigSupport Sub-task Resolved Hyukjin Kwon
              24.
              		 Avoid to create a readsupport at write path in Data Source V2 Sub-task Resolved Hyukjin Kwon
              25.
              		 data source v2 API improvement Sub-task Resolved Wenchen Fan

                Activity

                  People

                  • Assignee:
                    Unassigned
                    Reporter:
                    cloud_fan Wenchen Fan
                  • Votes:
                    6 Vote for this issue
                    Watchers:
                    28 Start watching this issue

                    Dates

                    • Created:
                      Updated: