Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25241

Configurable empty values when reading/writing CSV files

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.3.1, 2.4.0
    • Fix Version/s: 2.4.0
    • Component/s: SQL
    • Labels:
      None

      Description

      There is anĀ option in the CSV parser to set values when we have empty values in the CSV files or in our dataframes.

      Currently, this option cannot be configured and always sets a default value (empty string for reading and `""` for writing).

      I think it'd be interesting to enable this option in the CSV reader/writer to allow customizing these empty values.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mmolimar Mario Molina
                Reporter:
                mmolimar Mario Molina
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: