[SPARK-39489] Improve EventLoggingListener and ReplayListener performance by replacing Json4S ASTs with Jackson trees

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.4.0
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 16/Jun/22 01:53
• Updated: 14/Nov/23 21:10
• Resolved: 03/Jul/22 20:50

Description

Spark's event log JsonProtocol currently uses Json4s ASTs to generate and parse JSON. Performance overheads from Json4s account for a significant proportion of all time spent in JsonProtocol. If we replace Json4s usage with direct usage of Jackson APIs then we can significantly improve performance (~2x improvement for writing and reading in my own local microbenchmarks).

This performance improvement translates to faster history server load times and reduced load on the Spark driver (and reduced likelihood of dropping events because the listener cannot keep up, therefore reducing the likelihood of inconsistent Spark UIs).

Reducing our usage of Json4s is also a step towards being able to eventually remove our dependency on Json4s: Spark's current use of Json4s creates library conflicts for end users who want to adopt Json4s 4 (see discussion on PRs for SPARK-36408). If Spark can eventually remove its Json4s dependency then we will completely eliminate such conflicts.

Issue Links

• causes: SPARK-42403 JsonProtocol should handle null JSON strings
• links to: [Github] Pull Request #36885 (JoshRosen)
• links to: GitHub Pull Request #36885