Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-28307

FLIP-241: Completed Jobs Information Enhancement

    XMLWordPrintableJSON

Details

    • Hide
      We have enhanced the experiences of viewing completed jobs’ information in this release.
      - JobManager / HistoryServer WebUI now provides detailed execution time metrics, including durations tasks spend in each execution state and the accumulated busy / idle / back-pressured time during running.
      - JobManager / HistoryServer WebUI now provides aggregation of major SubTask metrics, grouped by Task or TaskManager.
      - JobManager / HistoryServer WebUI now provides more environmental information, including environment variables, JVM options and classpath.
      - HistoryServer now supports browsing logs from external log archiving services. For more details: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/advanced/historyserver#log-integration
      Show
      We have enhanced the experiences of viewing completed jobs’ information in this release. - JobManager / HistoryServer WebUI now provides detailed execution time metrics, including durations tasks spend in each execution state and the accumulated busy / idle / back-pressured time during running. - JobManager / HistoryServer WebUI now provides aggregation of major SubTask metrics, grouped by Task or TaskManager. - JobManager / HistoryServer WebUI now provides more environmental information, including environment variables, JVM options and classpath. - HistoryServer now supports browsing logs from external log archiving services. For more details: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/advanced/historyserver#log-integration

    Description

      Streaming and Batch users have different interests in probing a job. While streaming users mainly care about the instant status of a running job (tps, delay, backpressure, etc.), batch users care more about the overall job status during the entire execution (queueing / execution time, total data amount, etc.).

      As Flink grows into a unified streaming & batch processor and is adopted by more and more batch users, the experiences in inspecting completed jobs has become more important than ever.

      We compared Flink with other popular batch processors, and spotted several potential improvements. Most of these changes involves WebUI & REST API changes, which should be discussed and voted on as FLIPs. However, creating separated FLIPs for each of the improvement might be overkill, because changes needed by each improvement are quite small. Thus, we include all these potential improvements in this one FLIP.

      Attachments

        Issue Links

          There are no Sub-Tasks for this issue.

          Activity

            People

              Unassigned Unassigned
              guoyangze Yangze Guo
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: