Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41284

Feature parity: I/O in Spark Connect

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Reopened
    • Critical
    • Resolution: Unresolved
    • 3.4.0
    • None
    • Connect
    • None

    Description

      Implement I/O API such as DataFrameReader/Writer

      Attachments

        Issue Links

          split from

          Sub-task - The sub-task of the issue SPARK-39674 Initial protobuf definition for Spark Connect API

          • Major - Major loss of function.
          • Resolved
          1.
          		 PySpark read API parity for Spark Connect Sub-task Resolved Rui Wang
          2.
          		 PySpark write API for Spark Connect Sub-task Resolved Hyukjin Kwon
          3.
          		 Add basic support for DataFrameWriter Sub-task Resolved Martin Grund
          4.
          		 Unset Read.schema is incorrectly read when unset Sub-task Resolved Martin Grund
          5.
          		 Implement DataFrameReader.json Sub-task Resolved Rui Wang
          6.
          		 Implement DataFrameReader.parquet Sub-task Resolved Hyukjin Kwon
          7.
          		 Implement DataFrameReader.text Sub-task Resolved Sandeep Singh
          8.
          		 Add the unsupported function list Sub-task Resolved Ruifeng Zheng
          9.
          		 Support DataFrameWriter.saveAsTable Sub-task Resolved Takuya Ueshin
          10.
          		 SparkSession.read support reading with schema Sub-task Resolved Sandeep Singh
          11.
          		 NPE for bucketed write (ReadwriterTests.test_bucketed_write) Sub-task Resolved Takuya Ueshin
          12.
          		 saveAsTable fail to find the default source (ReadwriterTests.test_insert_into) Sub-task Resolved Takuya Ueshin
          13.
          		 Unexpected schema set to DefaultSource plan (ReadwriterTests.test_save_and_load) Sub-task Resolved Ruifeng Zheng
          14.
          		 Implement DataFrameWriterV2 (ReadwriterV2Tests) Sub-task Resolved Sandeep Singh
          15.
          		 Implement DataFrameReader.csv Sub-task Resolved Sandeep Singh
          16.
          		 Implement DataFrameReader.orc Sub-task Resolved Sandeep Singh
          17.
          		 Implement DataFrameReader.text to take multiple paths Sub-task Resolved Ruifeng Zheng
          18.
          		 DataFrameReader should support list of paths Sub-task Resolved Ruifeng Zheng
          19.
          		 DataFrameReader should support StructType schema Sub-task Resolved Ruifeng Zheng
          20.
          		 insertInto fails when the column names are different from the table columns Sub-task Resolved Takuya Ueshin
          21.
          		 Fix DataFrameWriterV2 to find the default source Sub-task Resolved Takuya Ueshin
          22.
          		 Fix DataFrameReader to use the default source Sub-task Resolved Takuya Ueshin
          23.
          		 DataFrame.toPandas should handle duplicated column names Sub-task Resolved Takuya Ueshin
          24.
          		 df.write.format().save() should support calling with no path or table name Sub-task Resolved Unassigned
          25.
          		 Implement DataFrameReader/Writer.jdbc Sub-task Resolved Takuya Ueshin

          Activity

            People

              amaliujia Rui Wang
              gurwls223 Hyukjin Kwon
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: