Implementing a Resource Manager (V1) for Hadoop

• Key: HADOOP-3444
• Type: New Feature
• Status: Open
• Priority: Major
• Assignee: Unassigned
• Reporter: Vivek Ratan

Description

HADOOP-3421 lists the requirements for a Resource Manager for Hadoop. This Jira tracks its implementation.

It is expected that this Jira will be used to keep track of various other Jiras that will be opened towards implementing Version 1 of the Resource Manager.

Comments

[Vivek Ratan]

RMArch-V1.jpg is a high level architecture.

It is proposed that today's JT be extended to support queues of jobs (actually, JobInProgress objects). The JobQueueManager handles these queues, which are persisted to disk. Users can interact with the queues, which are configured appropriately.

The JT today handles at least three separate functionalities: scheduling, monitoring, and Web serving. It is likely that in the future, these three are separated out, at least in terms of components/modules/classes within JT, if not outside it.

[Vivek Ratan]

My guess is, we'll need separate Jiras to track at least the following:

• the core scheduling component (the part of the new JT that includes the JobQueueManager and handles scheduling, guaranteed capacities for Orgs, user limits, etc.).
• configuration for Orgs, queues, users,quotas, limits, etc.
• Security, access control
• scalability/availability
• user interaction with the job queues (through Web, CLI)
• accounting

others?

[Vivek Ratan]

HADOOP-3445 tracks the implementation of the core scheduling component.