[REEF-1776] Use YARN proxy user to register REEF Driver in Unmanaged AM mode

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16
• Component/s: REEF Client, REEF Driver, REEF-Runtime-YARN
• Labels: yarn yarn-client
• Assignee: Sergiy Matusevych
• Reporter: Sergiy Matusevych
• Created: 13/Apr/17 21:15
• Updated: 18/Apr/17 23:57
• Resolved: 17/Apr/17 22:32

Description

When launching REEF application in Unmanaged AM mode from another YARN application, two Drivers access the same global object UserGroupInformation from YARN API. That issue prevents us from running REEF applications on YARN in REEF-on-REEF or REEF-on-Spark settings. We have to implement a mechanism to create proxy YARN users and register Unmanaged AM Drivers in that proxy user context and update the API so that REEF client can pass the user credentials into the driver setup.

Issue Links

• is related to: REEF-1667 REEF-on-REEF example
• is related to: REEF-1747 Workaround to allow two AMs in one YARN container
• relates to: YARN-5531 UnmanagedAM pool manager for federating application across clusters