[SPARK-37340] Display StageIds in Operators for SQL UI

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.0
• Component/s: Web UI
• Assignee: Unassigned
• Reporter: Yian Liou
• Created: 15/Nov/21 22:45
• Updated: 16/Nov/21 19:47

Description

This proposes a more generalized solution of https://issues.apache.org/jira/browse/SPARK-30209, where a stageId-> operator mapping is done with the following algorithm.

 1. Read SparkGraph to get every Node's name and respective AccumulatorIDs.
 2. Gets each stage's AccumulatorIDs.
 3. Maps Operators to stages by checking for non-zero intersection of Step 1 and 2's AccumulatorIDs.
 4. Connect SparkGraphNodes to respective StageIDs for rendering in SQL UI.

As a result, some operators without max metrics values will also have stageIds in the UI. This Jira also aims to add minor enhancements to the SQL UI tab.

Issue Links

• links to: [Github] Pull Request #34622 (yliou)