[SPARK-12420] Have a built-in CSV data source implementation

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: SQL
• Target Version/s: 2.0.0
• Assignee: Unassigned
• Reporter: Reynold Xin
• Created: 18/Dec/15 09:09
• Updated: 12/Dec/22 18:11
• Resolved: 15/Jul/16 21:59

Description

CSV is the most common data format in the "small data" world. It is often the first format people want to try when they see Spark on a single node. Making this built-in for the most common source can provide a better experience for first-time users.

We should consider inlining https://github.com/databricks/spark-csv

Issue Links

• is blocked by: SPARK-15840 New csv reader does not "determine the input schema"
• links to: [Github] Pull Request #10615 (falaki)

Sub-Tasks

• Initial import of databricks/spark-csv: Resolved
• Renaming CSV options to be similar to Pandas and R: Resolved
• Organize options for default values: Closed
• Use spark internal utilities wherever possible: Closed
• Improve tests for better coverage: Closed
• Populate statistics for DataFrame when reading CSV: Closed
• Support to specify the option for compression codec.: Resolved
• Refector options to be correctly formed in a case class: Resolved
• CSVRelation should be based on HadoopFsRelation: Closed
• Use cast expression to perform type cast in csv: Closed
• Encoding not working with non-ascii compatible encodings (UTF-16/32 etc.): Closed
• Expose maxCharactersPerColumn as a user configurable option: Resolved
• Documentation for CSV datasource options: Resolved
• Support for loading CSV with a single function call: Resolved
• NullPoingException in schema inference for CSV when the first line is empty: Resolved
• java.lang.NegativeArraySizeException in CSV: Resolved
• Make type inference recognize boolean types: Resolved
• Support for writing CSV with a single function call: Resolved
• Support for saving with a quote mode: Resolved
• Support for specifying custom date format for date and timestamp types: Resolved
• Keep old data source name for backwards compatibility: Resolved
• Limit logging of bad records: Resolved
• Handle decimal type in CSV inference: Resolved
• Produce InternalRow instead of external Row: Resolved
• Options for parsing NaNs, Infinity and nulls for numeric types: Resolved
• Increase default value for maxCharsPerColumn: Resolved
• rowSeparator does not work for both reading and writing: Resolved
• Put CSV options as Python csv function parameters: Resolved
• Upgrade Univocity library from 2.0.2 to 2.1.0: Resolved
• Allow setting the quoteEscapingEnabled flag when writing CSV: Resolved