[YARN-913] Umbrella: Add a way to register long-lived services in a YARN cluster

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.5.0, 2.4.1
• Component/s: api, resourcemanager
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 10/Jul/13 18:58
• Updated: 08/Jul/19 13:26

Description

In a YARN cluster you can't predict where services will come up -or on what ports. The services need to work those things out as they come up and then publish them somewhere.

Applications need to be able to find the service instance they are to bond to -and not any others in the cluster.

Some kind of service registry -in the RM, in ZK, could do this. If the RM held the write access to the ZK nodes, it would be more secure than having apps register with ZK themselves.

Issue Links

• is depended upon by: SLIDER-365 slider "resolve" command to retrieve service record
• is depended upon by: YARN-896 Roll up for long-lived services in YARN
• is depended upon by: SLIDER-149 Support a YARN service registry
• is related to: HADOOP-11117 UGI HadoopLoginModule doesn't catch & wrap all kerberos-related exceptions
• is related to: HADOOP-11111 MiniKDC to use locale EN_US for case conversions
• relates to: YARN-4758 Enable discovery of AMs by containers
• relates to: YARN-6413 FileSystem based Yarn Registry implementation
• relates to: YARN-1143 Restrict the names that apps and types can have

Sub-Tasks

• RM to support YARN registry: Resolved
• Add CLI client to the registry to list, view and manipulate entries: Closed
• distributed shell & tests to use registry: Open
• add hadoop-yarn-registry package under hadoop-yarn: Closed
• MiniYARNCluster to (optionally) add MicroZookeeper service: Resolved
• yarn-registry JAR won't link against ZK 3.4.5: Closed
• registry punycoding of usernames doesn't fix all usernames to be DNS-valid: Closed
• Improved Yarn Registry service record structure: Closed
• registry config options: document and move to core-default: Closed
• TestSecureRMRegistryOperations failing on windows: secure ZK won't start: Closed
• ktutil test hanging on some machines/ktutil versions: Closed
• TestSecureRMRegistryOperations failing on windows: auth problems: Closed
• Provide Read Write REST view of YARN registry and client compatible with existing Registry API: Open
• TTL for YARN Registry SRV records: Open
• Collision-free unique bindings & refresh APIs for service records: Open
• TTL & identity aware read cache for the SRV records: Open
• clean up registry code for java 7/8: Resolved
• YARN registry service should avoid scanning whole ZK tree for every container/application finish: Resolved