[SPARK-27019] Spark UI's SQL tab shows inconsistent values

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.1, 3.0.0
• Component/s: SQL, Web UI
• Assignee: shahid
• Reporter: peay
• Created: 01/Mar/19 11:08
• Updated: 25/Apr/20 11:57
• Resolved: 06/Mar/19 22:09

Description

Since 2.4.0, I am frequently seeing broken outputs in the SQL tab of the Spark UI, where submitted/duration make no sense, description has the ID instead of the actual description.

Clicking on the link to open a query, the SQL plan is missing as well.

I have tried to increase `spark.scheduler.listenerbus.eventqueue.capacity` to very large values like 30k out of paranoia that we may have too many events, but to no avail. I have not identified anything particular that leads to that: it doesn't occur in all my jobs, but it does occur in a lot of them still.

Issue Links

• relates to: SPARK-31564 Flaky test: AllExecutionsPageSuite.SPARK-27019:correctly display SQL page when event reordering happens
• links to: GitHub Pull Request #23939