[FLINK-28307] FLIP-241: Completed Jobs Information Enhancement

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Done
• Fix Version/s: 1.16.0
• Component/s: Runtime / REST, Runtime / Web Frontend
• Labels: pull-request-available

Release Note

We have enhanced the experiences of viewing completed jobs’ information in this release.
- JobManager / HistoryServer WebUI now provides detailed execution time metrics, including durations tasks spend in each execution state and the accumulated busy / idle / back-pressured time during running.
- JobManager / HistoryServer WebUI now provides aggregation of major SubTask metrics, grouped by Task or TaskManager.
- JobManager / HistoryServer WebUI now provides more environmental information, including environment variables, JVM options and classpath.
- HistoryServer now supports browsing logs from external log archiving services. For more details: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/advanced/historyserver#log-integration

• Assignee: Unassigned
• Reporter: Yangze Guo
• Created: 30/Jun/22 06:08
• Updated: 19/Jul/22 05:21
• Resolved: 19/Jul/22 05:21

Description

Streaming and Batch users have different interests in probing a job. While streaming users mainly care about the instant status of a running job (tps, delay, backpressure, etc.), batch users care more about the overall job status during the entire execution (queueing / execution time, total data amount, etc.).

As Flink grows into a unified streaming & batch processor and is adopted by more and more batch users, the experiences in inspecting completed jobs has become more important than ever.

We compared Flink with other popular batch processors, and spotted several potential improvements. Most of these changes involves WebUI & REST API changes, which should be discussed and voted on as FLIPs. However, creating separated FLIPs for each of the improvement might be overkill, because changes needed by each improvement are quite small. Thus, we include all these potential improvements in this one FLIP.

Issue Links

• links to: GitHub Pull Request #20297

Sub-Tasks

• Introduce accumulated time metrics that a running task is busy / idle / back-pressured: Closed
• Introduce metrics of the duration that a task stays in each status: Closed
• Introduce aggregating task metrics: Closed
• Introduce REST APIs for the environmental information: Closed
• Introduce REST APIs for log URL retrieval: Closed
• Add history server flag to DashboardConfiguration: Closed
• [UI] Introduce "Cluster Environment" tab under history server: Closed
• [UI] Introduce aggregate stats in tables of the subtasks and taskmanagers: Closed
• [UI] add external JM and TM log links under history server: Closed
• Update history server documentation: Closed