[YARN-7054] Yarn Service Phase 2

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: yarn-native-services
• Assignee: Jian He
• Reporter: Jian He
• Created: 19/Aug/17 05:17
• Updated: 08/Oct/18 18:07

Issue Links

• incorporates: YARN-7215 REST API to list all deployed services by the same user
• incorporates: YARN-7216 Missing ability to list configuration vs status
• incorporates: YARN-7472 Inconsistent annotation and getter/setter method for Service data model
• incorporates: YARN-7399 Yarn services metadata storage improvement
• incorporates: YARN-7217 Improve API service usability for updating service spec and state
• is related to: YARN-4793 [Umbrella] Simplified API layer for services and beyond
• is related to: YARN-4757 [Umbrella] Simplified discovery of services via DNS mechanisms
• is related to: YARN-5079 [Umbrella] Native YARN framework layer for services and beyond

Sub-Tasks

• Applications API Service HA: Resolved
• Auto-generate REST API resources and server side stubs from swagger definition: Open
• Provide a flag in Rest API GET response to notify if the app launch delay is due to docker image download.: Open
• Embed REST API service into RM: Resolved
• REST API to return the no of requested containers for apps in ACCEPTED state.: Open
• RM option to delegate NM loss container action to AM: Patch Available
• Restarted RM may not inform AM about all existing containers: Resolved
• Create a API class for yarn-native-service user-facing constants: Resolved
• Support kerberos deployment for yarn-native-service: Resolved
• Service REST API should have a way to specify YARN related params: Resolved
• Support specifying YARN related params in the service REST API: Resolved
• Rethink on configurations published into ATSv2: Open
• Allow service to be started as System Services during serviceapi start up: Open
• Support security for YARN service framework: Resolved
• Recover component information on YARN native services AM restart: Resolved
• Revisit deleteService call once YARN is able to do post clean up for an app: Open
• need configuration mechanism for specifying per-host network interface: Open
• Evaluate SLIDER-1185 (container/application diagnostics) wrt ATSv2 and Yarn Logging enhancements: Open
• Support placement policy in yarn native services: Resolved
• Add examples in yarn-service: Resolved
• Add an apache httpd example YARN service: Resolved
• Add UT for api-server: Resolved
• Some NPE fixes in Registry DNS: Resolved
• REST API to list all deployed services by the same user: Open
• Missing ability to list configuration vs status: In Progress
• ApiServer REST API naming convention /ws/v1 is already used in Hadoop v2: Resolved
• Add security check for privileged docker container: Resolved
• Data structure update in service REST API: Resolved
• Add recursion support and configure RegistryDNS to lookup upstream: Resolved
• Change in few metrics in new YARN UI related to native-services: Resolved
• Added allocateRequestId in NMContainerStatus for recovery: Resolved
• Remove apiserver cmd and merge service cmd into application cmd: Resolved
• RegistryDNS doesn't work in tcp channel: Resolved
• Improve service client loggings: Resolved
• Optimization to AM recovery when the service record doesn't exist for a container: Open
• Support placement policy in yarn service: Resolved
• TestYarnNativeServices testRecoverComponentsAfterRMRestart fails due to BindingException: Open
• Race condition in service AM that can cause NPE: Resolved
• Make ServiceClient.getAppId method public to return ApplicationId for a service name: Resolved
• Configurable heap size / JVM opts in service AM: Resolved
• Separate log file for service rest API: Open
• RegistryDNS does not work with secure ZooKeeper: Resolved
• Add --debug option for yarn service framework: Open
• TestYarnNativeServices#testRecoverComponentsAfterRMRestart() fails intermittently: Resolved
• hadoop-yarn-services-api should be part of hadoop-yarn-services: Resolved
• Convert yarn app cli to call yarn api services: Resolved
• FileNotFoundException due to a broken link when creating a yarn service and missing max cpu limit check: Resolved
• Yarn service pre-maturely releases the container after AM restart: Resolved
• Make the service status output more readable: In Progress
• Support resource profiles and fine-grained resource requests in native services: Resolved
• Implement doAs for Api Service REST API: Resolved
• Ensure long running service AM works in secure cluster: Resolved
• App status does not return state STABLE for a running and stable service: Resolved
• Can not destroy yarn service application in stopped state: Resolved
• Document improvement for registry dns: Resolved
• yarn application status should support application name: Resolved
• Update YARN-Services-Examples.md to be in sync with the latest code: Resolved
• Service AM should use configured default docker network: Resolved
• YARN Service component update PUT API should not use component name from JSON body: Resolved
• YARN Service - Two different users are unable to launch a service of the same name: Resolved
• Enable YARN service by default: Open
• Provide improved error message when YARN service is disabled: Resolved
• Add default service AM Xmx: Resolved
• Service AM gets NoAuth with secure ZK: Resolved
• Transitive dependencies for component are not resolved: Resolved
• Component status stays "Ready" when yarn service is stopped: Resolved
• Calling stop on an already stopped service says "Successfully stopped service": Resolved
• Improve status response when yarn application is destroyed: Resolved
• Allow user supplied Docker client configurations with YARN native services: Resolved
• Validate the application ID has been persisted to the service definition prior to use: Resolved
• Determine the appropriate default ContainerRetryPolicy: Resolved
• Support auto-spawning of admin configured services during bootstrap of rm/apiserver: Resolved
• Create default readiness check for service components: Resolved
• Clean up example hostnames: Resolved
• Yarn Service API site doc broken due to unwanted character in YarnServiceAPI.md: Resolved
• Support placement policy composite constraints in YARN Service: Open
• Component health threshold monitor: Resolved
• Support auto-spawning of admin configured services during bootstrap of RM: Resolved
• ServiceState FLEX should be removed: Open
• Introduce container readiness check type: Patch Available
• Provide functional artifact & launch_command in examples of YARN-Services-Examples.md: Open
• Flex down should remove instance with largest component instance ID first: Resolved
• Remove YARN registry entries when a service is killed by the RM: Resolved
• Enable stopped system services to be started during RM start: Resolved
• Yarn service: revisit the version in service: Open
• Yarn Service Destroy: Delete service entries from Zookeeper in the ServiceMaster instead of ServiceClient in the RM: Open
• Move overwriting of service definition during flex to service master: Open
• YARN service example is out-dated: Resolved