[SPARK-42693] API Auditing

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Done
• Affects Version/s: 3.4.0
• Component/s: ML, PySpark, Spark Core, SQL, Structured Streaming
• Target Version/s: 3.4.0
• Assignee: Xinrong Meng
• Reporter: Xinrong Meng
• Created: 07/Mar/23 07:47
• Updated: 18/Jan/24 02:02
• Resolved: 30/Mar/23 01:18

Description

Audit user-facing API of Spark 3.4. The main goal is to ensure public API docs to be ready for release, for example, no private classes/methods is leaking and marked public.

There are 3 common ways to audit API:

• build docs (into a local website) against branch-3.4 to check
• 'git diff' to check the source code differences between v3.3.2 and branch-3.4
• https://github.com/apache/spark-website/pull/443 shows most of the API doc differences between v3.3.2 and the 3.4.0 RC4(the latest RC); commits are categorized by components

Issue Links

• is a child of: SPARK-42523 Apache Spark 3.4 release