Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41284

Feature parity: I/O in Spark Connect

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Reopened
    • Critical
    • Resolution: Unresolved
    • 3.4.0
    • None
    • Connect
    • None

    Description

      Implement I/O API such as DataFrameReader/Writer

      Attachments

        Issue Links

          1.
          PySpark read API parity for Spark Connect Sub-task Resolved Rui Wang
          2.
          PySpark write API for Spark Connect Sub-task Resolved Hyukjin Kwon
          3.
          Add basic support for DataFrameWriter Sub-task Resolved Martin Grund
          4.
          Unset Read.schema is incorrectly read when unset Sub-task Resolved Martin Grund
          5.
          Implement DataFrameReader.json Sub-task Resolved Rui Wang
          6.
          Implement DataFrameReader.parquet Sub-task Resolved Hyukjin Kwon
          7.
          Implement DataFrameReader.text Sub-task Resolved Sandeep Singh
          8.
          Add the unsupported function list Sub-task Resolved Ruifeng Zheng
          9.
          Support DataFrameWriter.saveAsTable Sub-task Resolved Takuya Ueshin
          10.
          SparkSession.read support reading with schema Sub-task Resolved Sandeep Singh
          11.
          NPE for bucketed write (ReadwriterTests.test_bucketed_write) Sub-task Resolved Takuya Ueshin
          12.
          saveAsTable fail to find the default source (ReadwriterTests.test_insert_into) Sub-task Resolved Takuya Ueshin
          13.
          Unexpected schema set to DefaultSource plan (ReadwriterTests.test_save_and_load) Sub-task Resolved Ruifeng Zheng
          14.
          Implement DataFrameWriterV2 (ReadwriterV2Tests) Sub-task Resolved Sandeep Singh
          15.
          Implement DataFrameReader.csv Sub-task Resolved Sandeep Singh
          16.
          Implement DataFrameReader.orc Sub-task Resolved Sandeep Singh
          17.
          Implement DataFrameReader.text to take multiple paths Sub-task Resolved Ruifeng Zheng
          18.
          DataFrameReader should support list of paths Sub-task Resolved Ruifeng Zheng
          19.
          DataFrameReader should support StructType schema Sub-task Resolved Ruifeng Zheng
          20.
          insertInto fails when the column names are different from the table columns Sub-task Resolved Takuya Ueshin
          21.
          Fix DataFrameWriterV2 to find the default source Sub-task Resolved Takuya Ueshin
          22.
          Fix DataFrameReader to use the default source Sub-task Resolved Takuya Ueshin
          23.
          DataFrame.toPandas should handle duplicated column names Sub-task Resolved Takuya Ueshin
          24.
          df.write.format().save() should support calling with no path or table name Sub-task Resolved Unassigned
          25.
          Implement DataFrameReader/Writer.jdbc Sub-task Resolved Takuya Ueshin

          Activity

            People

              amaliujia Rui Wang
              gurwls223 Hyukjin Kwon
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: