[SPARK-30936] Forwards-compatibility in JsonProtocol in broken

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Shixiong Zhu
• Reporter: Shixiong Zhu
• Created: 24/Feb/20 06:52
• Updated: 25/Feb/20 04:36
• Resolved: 25/Feb/20 04:36

Description

JsonProtocol is supposed to provide strong backwards-compatibility and forwards-compatibility guarantees: any version of Spark should be able to read JSON output written by any other version, including newer versions.

However, the forwards-compatibility guarantee is broken for events parsed by "ObjectMapper". If a new field is added to an event parsed by "ObjectMapper" (e.g., https://github.com/apache/spark/commit/6dc5921e66d56885b95c07e56e687f9f6c1eaca7#diff-dc5c7a41fbb7479cef48b67eb41ad254R33), this event cannot be parsed by an old version of Spark History Server.

Issue Links

• links to: GitHub Pull Request #27680