[FLINK-6042] Display last n exceptions/causes for job restarts in Web UI

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.13.0
• Component/s: Runtime / Coordination, Runtime / Web Frontend
• Labels: pull-request-available

Release Note

Flink exposes the exception history now through the REST API and the UI. The amount of most-recently handled exceptions that shall be tracked can be defined through `web.exception-history-size`. Some values of the exception history's REST API Json response are deprecated as part of this effort.

• Assignee: Matthias Pohl
• Reporter: Till Rohrmann
• Created: 14/Mar/17 15:23
• Updated: 21/Apr/21 10:08
• Resolved: 20/Apr/21 16:33

Description

Users requested that it would be nice to see the last n exceptions causing a job restart in the Web UI. This will help to more easily debug and operate a job.

We could store the root causes for failures similar to how prior executions are stored in the ExecutionVertex using the EvictingBoundedList and then serve this information via the Web UI.

-- Update: January 21, 2021 --

The UI can already handle multiple exceptions through the Exception History. Right now, we list one or more exceptions which caused the job to fail. Instead, we could adapt it in a way that the history contains not only the exceptions of the most recent failure but one expandable entry per restart. If there are more than one exception connected to a single restart, we would list their stacktraces within one expandable entry.

Issue Links

• is duplicated by: FLINK-12662 Record failure and restart information for ExecutionGraph to keep history information
• relates to: FLINK-14143 Failed Attempts display in the timeline
• relates to: FLINK-22144 Test display last n exceptions/causes for job restarts in Web UI
• relates to: FLINK-21439 Adaptive Scheduler: Add support for exception history
• links to: GitHub Pull Request #3601
• links to: GitHub Pull Request #3643
• links to: GitHub Pull Request #14686
• mentioned in: Page Loading...

Sub-Tasks

• RootException history implementation in DefaultScheduler: Closed
• Introduce wrapper class to hold information besides ArchivedExecutionGraph: Closed
• Extend exception history to collect concurrent task failure that have a common root cause: Closed
• Expose extended exception history through REST handler and adapt UI accordingly: Closed