[YARN-7292] Retrospect Resource Profile Behavior for overriding capability

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 3.1.0
• Component/s: nodemanager, resourcemanager
• Target Version/s: 3.1.0, 3.2.0
• Hadoop Flags: Reviewed
• Assignee: Wangda Tan
• Reporter: Wangda Tan
• Created: 06/Oct/17 00:16
• Updated: 16/Feb/18 08:33
• Resolved: 16/Feb/18 07:08

Description

Had discussions with templedf, vvasudev, sunilg offline. There're a couple of resource profile related behaviors might need to be updated:

1) Configure resource profile in server side or client side:

Currently resource profile can be only configured centrally:

• Advantages:
  A given resource profile has a the same meaning in the cluster. It won’t change when we run different apps in different configurations. A job can run under Amazon’s G2.8X can also run on YARN with G2.8X profile. A side benefit is YARN scheduler can potentially do better bin packing.
• Disadvantages:
  Hard for applications to add their own resource profiles.

2) Do we really need mandatory resource profiles such as minimum/maximum/default?

3) Should we send resource profile name inside ResourceRequest, or should client/AM translate it to resource and set it to the existing resource fields?

4) Related to above, should we allow resource overrides or client/AM should send final resource to RM?