[FLINK-15679] Improve Flink's ID system

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Done
• Fix Version/s: 1.12.0
• Component/s: Runtime / Coordination
• Labels: usability
• Assignee: Yangze Guo
• Reporter: Yangze Guo
• Created: 20/Jan/20 06:01
• Updated: 14/Jan/21 09:26
• Resolved: 19/Aug/20 02:00

Description

Flink uses a lot of IDs which are hard to decipher for the user:

• The string literal of most Flink’s IDs are meaningless hashcode. It does not help the user to understand what happened.
• The LOGs do not contain the lineage info of those IDs. Currently, it’s impossible to track the end to end lifecycle of an Execution or a Task.
• Some redundancy exists in Flink’s ID system, we need to sort out the current ID system before introducing more complexity.

To help the user to understand what Flink is doing, we target to add more meaning to Flink’s IDs and LOGs while cleaning up the redundancy and refactoring the ID system.

This is an umbrella ticket to track all the relevant issues.

Issue Links

• is duplicated by: FLINK-9079 Job Id or Job Name in task manager logs
• relates to: FLINK-17331 Add NettyMessageContent interface for all the class which could be write to NettyMessage
• mentioned in: Page Loading...
• mentioned in: Page Loading...

Sub-Tasks

• Consider refactoring of IntermediateResultPartitionID to consist of IntermediateDataSetID and partitionIndex: Closed
• Refactor the ExecutionAttemptID to consist of ExecutionVertexID and attemptNumber: Closed
• Log the lineage information between ExecutionAttemptID and SlotRequestID: Closed
• Log the lineage information between SlotRequestID and AllocationID: Closed
• Log the lineage information between ExecutionAttemptID and AllocationID: Closed
• Log host informations for TaskManager failures.: Closed
• Set the resource id of taskexecutor according to environment variable if exist in standalone mode: Closed