[SPARK-18760] Provide consistent format output for all file formats

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.1.0
• Component/s: SQL
• Target Version/s: 2.2.0
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 07/Dec/16 05:21
• Updated: 15/Dec/16 05:05
• Resolved: 08/Dec/16 20:52

Description

We currently rely on FileFormat implementations to override toString in order to get a proper explain output. It'd be better to just depend on shortName for those.

Before:

scala> spark.read.text("test.text").explain()
== Physical Plan ==
*FileScan text [value#15] Batched: false, Format: org.apache.spark.sql.execution.datasources.text.TextFileFormat@xyz, Location: InMemoryFileIndex[file:/scratch/rxin/spark/test.text], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<value:string>

After:

scala> spark.read.text("test.text").explain()
== Physical Plan ==
*FileScan text [value#15] Batched: false, Format: text, Location: InMemoryFileIndex[file:/scratch/rxin/spark/test.text], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<value:string>

Issue Links

• duplicates: SPARK-17101 Provide consistent format identifiers for TextFileFormat and ParquetFileFormat
• links to: [Github] Pull Request #16187 (rxin)