[YARN-5079] [Umbrella] Native YARN framework layer for services and beyond

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.1.0
• Target Version/s: 3.1.0
• Release Note: A framework is implemented to orchestrate containers on YARN
• Assignee: Unassigned
• Reporter: Vinod Kumar Vavilapalli
• Created: 12/May/16 21:45
• Updated: 22/Mar/18 05:51
• Resolved: 06/Nov/17 22:55

Description

(See overview doc at YARN-4692, modifying and copy-pasting some of the relevant pieces and sub-section 3.3.1 to track the specific sub-item.)

(This is a companion to YARN-4793 in our effort to simplify the entire story, but focusing on APIs)

So far, YARN by design has restricted itself to having a very low-­level API that can support any type of application. Frameworks like Apache Hadoop MapReduce, Apache Tez, Apache Spark, Apache REEF, Apache Twill, Apache Helix and others ended up exposing higher level APIs that end­-users can directly leverage to build their applications on top of YARN. On the services side, Apache Slider has done something similar.

With our current attention on making services first­-class and simplified, it's time to take a fresh look at how we can make Apache Hadoop YARN support services well out of the box. Beyond the functionality that I outlined in the previous sections in the doc on how NodeManagers can be enhanced to help services, the biggest missing piece is the framework itself. There is a lot of very important functionality that a services' framework can own together with YARN in executing services end­-to­-end.

In this JIRA I propose we look at having a native Apache Hadoop framework for running services natively on YARN.

Issue Links

• incorporates: YARN-4757 [Umbrella] Simplified discovery of services via DNS mechanisms
• is part of: YARN-4692 [Umbrella] Simplified and first-class support for services in YARN
• is related to: YARN-4793 [Umbrella] Simplified API layer for services and beyond
• is related to: YARN-5780 [YARN-5079] Allowing YARN native services to post data to timeline service V.2
• relates to: YARN-7054 Yarn Service Phase 2
• links to: GitHub Pull Request #238

Sub-Tasks

• Port initial slider-core module code into yarn: Resolved
• Create an agent-less docker provider in the native-services framework: Resolved
• Move Java only tests from slider develop to yarn-native-services: Resolved
• Apply SLIDER-875 to yarn-native-services for supporting a composite app-of-apps: Resolved
• Apply SLIDER-1166 to yarn-native-services branch: Resolved
• Add 2 new fields in Slider status output - image-name and is-privileged-container: Resolved
• Integrate native services modules into maven build: Resolved
• Fix issues in yarn native services apps-of-apps: Resolved
• Add a new field in Slider status output - lifetime (remaining): Resolved
• Add .keep file for yarn native services AM web app: Resolved
• Slider version output includes variables buildNumber and hadoop.version: Resolved
• Native services client errors out when config formats are uppercase: Resolved
• Remove the agent code in slider: Resolved
• Slider handles "per.component" for multiple components incorrectly: Resolved
• Write native services container stderr file to log directory: Resolved
• Native services AM should remain up if RM is down: Resolved
• Add Slider debug config in yarn log4j: Resolved
• Fix ASF license warnings for slider core module: Resolved
• Generate native services protobuf classes during build: Resolved
• Fix slider core module findbugs warnings: Resolved
• Fix slider core module javadocs: Resolved
• Remove the agent - slider AM ssl related code: Resolved
• Allow native services quicklinks to be exported for each component: Resolved
• Native services AM kills app on AMRMClientAsync onError call: Resolved
• Native services AM kills app on AMRMClientAsync onError call: Resolved
• Followup fix for slider core module findbugs: Resolved
• Create an agent-less docker-less provider in the native services framework: Resolved
• Apply SLIDER-1199 to yarn native services for blacklisting nodes: Resolved
• Handle InvalidResourceRequestException in native services AM onError: Resolved
• Move yarn-native-service API records into hadoop-yarn-slider-core module: Resolved
• Default lifetime for native services app is invalid: Resolved
• Refactor yarn-native-services framework: Resolved
• Port slider's groovy unit tests to yarn native services: Resolved
• Support service assembly: Resolved
• Integrate service app master to write data into ATSv2.: Resolved
• Support to add native-service specific details in new YARN UI: Resolved
• Remove some unneeded code after YARN-6255: Resolved
• Improve configuring services through REST API: Resolved
• Revisit ATSv2 integration to ensure all required information is published: Resolved
• Support to launch new native-service from new YARN UI: Resolved
• Race condition in writing service record to registry in yarn native services: Resolved
• Followup fix for YARN-6405: Resolved
• Add native-service AM log4j properties in classpath: Resolved
• Update json validation for new native services providers: Resolved
• Services API delete call first attempt usually fails: Resolved
• We should validate yarn native services application submission side to ensure that the hostname should be less than 63 characters: Resolved
• SliderAppMaster gets an NPE when creating UnregisterComponentInstance: Resolved
• Native services support for specifying component start order: Resolved
• Bumping up pom file hadoop version: Resolved
• Allow custom hostname for docker containers in native services: Resolved
• Yarn-native-service framework core rewrite: Resolved
• Add more UTs in yarn-native-services: Resolved
• Post cleanup after YARN-6903, removal of org.apache.slider package: Resolved
• Rename application to service in yarn-native-services: Resolved
• Clean up packaging and dependencies for yarn-native-services: Resolved
• Create introductory site documentation for YARN native services: Resolved
• Merge yarn-native-service branch into trunk: Resolved
• Create install doc for YARN native services: Resolved
• Miscellaneous fixes in yarn-native-services: Resolved
• Improve yarn-service documentation: Resolved
• Add jsvc support for RegistryDNS: Resolved
• Fix high CPU usage issue in RegistryDNS: Resolved