Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • Connect
    • None

    Description

                                                                                
      pyspark/sql/tests/test_datasources.py:147 (DataSourcesParityTests.test_checking_csv_header)
      self = <pyspark.sql.tests.connect.test_parity_datasources.DataSourcesParityTests testMethod=test_checking_csv_header>
      
          def test_checking_csv_header(self):
              path = tempfile.mkdtemp()
              shutil.rmtree(path)
              try:
                  self.spark.createDataFrame([[1, 1000], [2000, 2]]).toDF("f1", "f2").write.option(
                      "header", "true"
                  ).csv(path)
                  schema = StructType(
                      [
                          StructField("f2", IntegerType(), nullable=True),
                          StructField("f1", IntegerType(), nullable=True),
                      ]
                  )
                  df = (
      >               self.spark.read.option("header", "true")
                      .schema(schema)
                      .csv(path, enforceSchema=False)
                  )
      
      ../test_datasources.py:162: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      
      self = <pyspark.sql.connect.readwriter.DataFrameReader object at 0x7fb118289520>
      args = ('/var/folders/0c/q8y15ybd3tn7sr2_jmbmftr80000gp/T/tmp4kdxohcw',)
      kwargs = {'enforceSchema': False}
      
          def csv(self, *args: Any, **kwargs: Any) -> None:
      >       raise NotImplementedError("csv() is not implemented.")
      E       NotImplementedError: csv() is not implemented.
      
      ../../connect/readwriter.py:225: NotImplementedError
      
      

      Attachments

        Activity

          People

            techaddict Sandeep Singh
            gurwls223 Hyukjin Kwon
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: