Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25241

Configurable empty values when reading/writing CSV files

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.3.1, 2.4.0
    • 2.4.0
    • SQL
    • None

    Description

      There is anĀ option in the CSV parser to set values when we have empty values in the CSV files or in our dataframes.

      Currently, this option cannot be configured and always sets a default value (empty string for reading and `""` for writing).

      I think it'd be interesting to enable this option in the CSV reader/writer to allow customizing these empty values.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            mmolimar Mario Molina
            mmolimar Mario Molina
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment