[TEZ-753] [Umbrella] Scalability improvements

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Siddharth Seth
• Created: 21/Jan/14 21:10
• Updated: 24/Apr/15 22:23

Issue Links

• relates to: TEZ-2316 Log count of different events if eventQueue has lot of events
• relates to: TEZ-2317 Event processing backlog can result in task failures for short tasks
• relates to: TEZ-2325 Route status update event directly to the attempt
• relates to: TEZ-1897 Create a concurrent version of AsyncDispatcher

Sub-Tasks

• Optionally send user payload as local resource if user payload if large: Open
• Investigate : skip user paylaod for same-vertex tasks when container re-use is used: Open
• Avoid creating multiple copies of the same Event payload: Closed
• Re-use ID instances in the AM, intern vertex names etc where possible: Closed
• Reduce AM mem usage caused by storing TezEvents: Closed
• Avoid re-sending TaskContext more than once to the same node: Open
• Dynamic heartbeat intervals between RM and AM: Resolved
• Dynamic heartbeat intervals between task and AM: Open
• The AM-RM heartbeat interval should not be static: Reopened
• Offload DataMovement event creation from the AM to the tasks: Open