[REEF-1747] Workaround to allow two AMs in one YARN container

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Component/s: REEF Runtime YARN, REEF-Runtime-YARN
• Labels: workaround
• Assignee: Sergiy Matusevych
• Reporter: Sergiy Matusevych
• Created: 09/Mar/17 00:30
• Updated: 18/Apr/17 23:57
• Resolved: 18/Apr/17 23:57

Description

The current version of YARN Java client does not allow us to register two Application Masters running in the same process. Technically, YARN Resource Manager does not care which process each AM runs in. However, there is a problem with the YARN Java client implementation: this library contains a singleton UserGroupInformation object that holds the user credentials of the current RM session. This data structure is shared by all AMs, and when REEF application tries to register the second (unmanaged) AM, the client library presents to YARN RM all credentials, including the security token of the first (managed) AM. YARN rejects such registration request, throwing InvalidApplicationMasterRequestException "Application Master is already registered".

A proper fix for this issue would be a patch for Hadoop YARN Java client, that would allow us to pass the required security token into the AMRMClientAsync.registerApplicationMaster() call. We also need a quick workaround in REEF so we can run REEF-on-REEF and Spark+REEF applications using unpatched Hadoop libraries.

Issue Links

• relates to: REEF-1776 Use YARN proxy user to register REEF Driver in Unmanaged AM mode