Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12420 Have a built-in CSV data source implementation
  3. SPARK-13261

Expose maxCharactersPerColumn as a user configurable option

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0
    • SQL
    • None

    Description

      We are using Univocity parser in the CSV data source in Spark. The parser has a fairly small limit for maximum number of characters per column. Spark's CSV data source updates it but it is not exposed to user. There are still use cases where the limit is too small. I think we should just expose it as an option. I suggest "maxCharsPerColumn" for the option.

      Attachments

        Activity

          People

            falaki Hossein Falaki
            falaki Hossein Falaki
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: