Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-35805 API auditing in Pandas API on Spark
  3. SPARK-35806

Mapping the `mode` argument to pandas in DataFrame.to_csv

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0
    • 3.2.0
    • PySpark
    • None

    Description

      pandas and pandas-on-Spark both have an argument named `mode` in the DataFrame.to_csv, but the acceptable strings are different.

      pandas can accept the "w", "w+", "a", "a+" where as pandas-on-Spark can accept "append", "overwrite", "ignore", "error" or "errorifexists".

      We should map these acceptable strings to pandas.

      e.g. "w" will work as Spark's "overwrite". In addition, mode can take Spark's "overwrite" too.

      Attachments

        Activity

          People

            itholic Haejoon Lee
            itholic Haejoon Lee
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: