[SPARK-39636] Fix multiple small bugs in JsonProtocol, impacting StorageLevel and Task/Executor resource requests

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 3.4.0
• Component/s: Spark Core, Web UI
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 30/Jun/22 00:37
• Updated: 30/Jun/22 20:42
• Resolved: 30/Jun/22 20:42

Description

While working on SPARK-39489, I spotted three pre-existing bugs in JsonProtocol:

• TaskResourceRequest loses precision for values < 0.5. The amount is a floating point number which is either between 0 and 0.5 or is a positive integer, but the read path assumes it is an integer.
• ExecutorResourceRequest integer overflows for values larger than an integer because the write path writes longs but the read path assumes integers.
• The OFF_HEAP storage level is not handled properly: the offHeap field isn't included in the JSON, so this StorageLevel cannot be round-tripped through JSON. This could cause the History Server to display inaccurate "off heap memory used" stats on the executors page.

I'm submitting a separate PR to fix those issues.

Issue Links

• links to: [Github] Pull Request #37027 (JoshRosen)