[FLINK-1541] GSoC project: Distributed profiling

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Won't Fix
• Component/s: Runtime / Coordination, Runtime / Task
• Labels: distributed gsoc2015 java scala
• Assignee: Unassigned
• Reporter: Till Rohrmann
• Created: 13/Feb/15 16:06
• Updated: 30/Jan/17 15:48
• Resolved: 30/Jan/17 15:48

Description

Understanding why a given program behaves in a certain way is often difficult and requires in many cases profiling. The profiling allows to detect hot spots in a program which are worthwhile for further investigation. For non-distributed applications there exists a wide variety of useful tools which achieve exactly that. However, Flink currently lacks the capability to profile distributed jobs.

Therefore, we would like to set up the infrastructure necessary for distributed profiling. First, this requires a thorough evaluation of existing tools in order to see how they could be used for profiling within Flink.

The profiling will most likely comprises the following steps:

1. Initiating the system and user code instrumentation
2. Collecting the profiling information on the distributed components
3. Aggregation of the generated profiling information and a proper visualization