[TEZ-4358] Make memory usage provided by DagClient be extensible without further changes

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: László Bodor
• Reporter: László Bodor
• Created: 03/Dec/21 13:47
• Updated: 30/Jan/22 11:40

Description

By TEZ-4339, DagClient can expose the sum heap usage by AM and tasks, however, there were 2 separate API endpoints introduced in order to expose those. This can become very verbose in code sooner or later once we want to expose more advanced data. The point is that we don't want to extend the DagClient API every time when we add a single property that is related to the application's memory usage. Currently, protobuf doesn't support map, so a workaround is needed, e.g a prop,value,prop,value serialized list, like:

list: ["am_consumption",value,"tasks_consumption",value,"task_containers_usage",value1:value2:value3,"task_containers_usage_percentage",value1:value2:value3]

Issue Links

• is related to: TEZ-4339 Expose real-time memory consumption of AM and task containers via DagClient