[NEMO-433] Improvement of Task Metrics and Collecting Them For Sampling

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.3
• Labels: pull-request-available
• Assignee: Hwarim Hyun
• Reporter: Wonook
• Created: 15/Jan/20 07:51
• Updated: 25/Nov/20 15:46
• Resolved: 10/Mar/20 10:08

Description

Currently, task metrics in Nemo are limited to only a few metrics, such as serialized/encoded read bytes and state transitions. This should be refactored and improved in the way so that the metrics are aggregated and managed in a single object, and we should be able to sample tasks for deriving future durations

There are a few references we can refer to:
monitoring.html

and

core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala in the Spark source code

Issue Links

• Blocked: NEMO-441 Dynamic Task Sizing on Nemo
• links to: GitHub Pull Request #289