Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39689

Support 2-chars lineSep in CSV datasource

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.4.0
    • SQL
    • None

    Description

      Univocity parser allows to set line separator to 1 to 2 characters (code), CSV options should not block this usage (code).

       

      Due to the limitation around the `normalizedNewLine` (https://github.com/uniVocity/univocity-parsers/issues/170), setting 2 chars as a line separator could cause some weird/bad behaviors. Thus, we probably should leave this proposed fix as an undocumented feature and warn users to do this.

       

      A more proper fix could be further investigated in the future.

      Attachments

        Activity

          People

            chinatsui Yaohua Cui
            yaohua Yaohua Zhao
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: