Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18943

Avoid per-record type dispatch in CSV when reading

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.2.0
    • SQL
    • None

    Description

      This is similar with https://issues.apache.org/jira/browse/SPARK-16434

      CSVRelation.csvParser does type dispatch for each value in each row.
      We can prevent this because the schema is already kept in CSVRelation.

      So, converters can be created first according to the schema, and then apply them to each.

      Attachments

        Activity

          People

            hyukjin.kwon Hyukjin Kwon
            hyukjin.kwon Hyukjin Kwon
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: