[SPARK-39658] Reconsider exposure of Json4s symbols in public ResourceInformation.toJson API

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Josh Rosen
• Created: 02/Jul/22 00:18
• Updated: 25/Jun/23 02:54

Description

Followup to discussion at https://github.com/apache/spark/pull/36885#discussion_r898644147

Spark's `ResourceInformation` class exposes Json4s symbols in its public API (via the toJson method). 

In the long run, I don't think that exposing third-party library symbols in our public APIs is a good idea. Perhaps we should make this API private or change it to require that users / implementors return an object which can be turned to JSON via Jackson ObjectMapper.

In principle we still have some freedom to change this because the ResourceInformation class is marked as `@Evolving`

I'm filing this ticket so that I can reference it in TODO code comments.

Issue Links

• is related to: SPARK-44172 Use Jackson API Instead of Json4s