[SPARK-19721] Good error message for version mismatch in log files

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.1.1, 2.2.0
• Component/s: Structured Streaming
• Target Version/s: 2.2.0
• Assignee: Liwei Lin
• Reporter: Michael Armbrust
• Created: 24/Feb/17 01:46
• Updated: 17/Mar/17 17:41
• Resolved: 17/Mar/17 17:41

Description

There are several places where we write out version identifiers in various logs for structured streaming (usually v1). However, in the places where we check for this, we throw a confusing error message. Instead, we should do the following:

• Find all of the places where we do this kind of check.
• for vN where n>1 say "UnsupportedLogFormat: The file path was produced by a newer version of Spark and cannot be read by this version. Please upgrade"
• for anything else throw an error saying the file is malformed.

Issue Links

• links to: [Github] Pull Request #17070 (lw-lin)