Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5079

[Umbrella] Native YARN framework layer for services and beyond

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.1.0
    • None
    • None
    • A framework is implemented to orchestrate containers on YARN

    Description

      (See overview doc at YARN-4692, modifying and copy-pasting some of the relevant pieces and sub-section 3.3.1 to track the specific sub-item.)

      (This is a companion to YARN-4793 in our effort to simplify the entire story, but focusing on APIs)

      So far, YARN by design has restricted itself to having a very low-­level API that can support any type of application. Frameworks like Apache Hadoop MapReduce, Apache Tez, Apache Spark, Apache REEF, Apache Twill, Apache Helix and others ended up exposing higher level APIs that end­-users can directly leverage to build their applications on top of YARN. On the services side, Apache Slider has done something similar.

      With our current attention on making services first­-class and simplified, it's time to take a fresh look at how we can make Apache Hadoop YARN support services well out of the box. Beyond the functionality that I outlined in the previous sections in the doc on how NodeManagers can be enhanced to help services, the biggest missing piece is the framework itself. There is a lot of very important functionality that a services' framework can own together with YARN in executing services end­-to­-end.

      In this JIRA I propose we look at having a native Apache Hadoop framework for running services natively on YARN.

      Attachments

        Issue Links

          incorporates

          New Feature - A new feature of the product, which has yet to be developed. YARN-4757 [Umbrella] Simplified discovery of services via DNS mechanisms

          • Major - Major loss of function.
          • Resolved
          is part of

          New Feature - A new feature of the product, which has yet to be developed. YARN-4692 [Umbrella] Simplified and first-class support for services in YARN

          • Major - Major loss of function.
          • Reopened
          is related to

          Bug - A problem which impairs or prevents the functions of the product. YARN-4793 [Umbrella] Simplified API layer for services and beyond

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. YARN-5780 [YARN-5079] Allowing YARN native services to post data to timeline service V.2

          • Major - Major loss of function.
          • Resolved
          relates to

          New Feature - A new feature of the product, which has yet to be developed. YARN-7054 Yarn Service Phase 2

          • Major - Major loss of function.
          • Open
          links to

          Web Link GitHub Pull Request #238

          1.
          		 Port initial slider-core module code into yarn Sub-task Resolved Jian He
          2.
          		 Create an agent-less docker provider in the native-services framework Sub-task Resolved Billie Rinaldi
          3.
          		 Move Java only tests from slider develop to yarn-native-services Sub-task Resolved Gour Saha
          4.
          		 Apply SLIDER-875 to yarn-native-services for supporting a composite app-of-apps Sub-task Resolved Billie Rinaldi
          5.
          		 Apply SLIDER-1166 to yarn-native-services branch Sub-task Resolved Gour Saha
          6.
          		 Add 2 new fields in Slider status output - image-name and is-privileged-container Sub-task Resolved Billie Rinaldi
          7.
          		 Integrate native services modules into maven build Sub-task Resolved Billie Rinaldi
          8.
          		 Fix issues in yarn native services apps-of-apps Sub-task Resolved Billie Rinaldi
          9.
          		 Add a new field in Slider status output - lifetime (remaining) Sub-task Resolved Jian He
          10.
          		 Add .keep file for yarn native services AM web app Sub-task Resolved Billie Rinaldi
          11.
          		 Slider version output includes variables buildNumber and hadoop.version Sub-task Resolved Gour Saha
          12.
          		 Native services client errors out when config formats are uppercase Sub-task Resolved Billie Rinaldi
          13.
          		 Remove the agent code in slider Sub-task Resolved Jian He
          14.
          		 Slider handles "per.component" for multiple components incorrectly Sub-task Resolved Billie Rinaldi
          15.
          		 Write native services container stderr file to log directory Sub-task Resolved Billie Rinaldi
          16.
          		 Native services AM should remain up if RM is down Sub-task Resolved Billie Rinaldi
          17.
          		 Add Slider debug config in yarn log4j Sub-task Resolved Unassigned
          18.
          		 Fix ASF license warnings for slider core module Sub-task Resolved Billie Rinaldi
          19.
          		 Generate native services protobuf classes during build Sub-task Resolved Billie Rinaldi
          20.
          		 Fix slider core module findbugs warnings Sub-task Resolved Jian He
          21.
          		 Fix slider core module javadocs Sub-task Resolved Billie Rinaldi
          22.
          		 Remove the agent - slider AM ssl related code Sub-task Resolved Jian He
          23.
          		 Allow native services quicklinks to be exported for each component Sub-task Resolved Billie Rinaldi
          24.
          		 Native services AM kills app on AMRMClientAsync onError call Sub-task Resolved Billie Rinaldi
          25.
          		 Native services AM kills app on AMRMClientAsync onError call Sub-task Resolved Billie Rinaldi
          26.
          		 Followup fix for slider core module findbugs Sub-task Resolved Jian He
          27.
          		 Create an agent-less docker-less provider in the native services framework Sub-task Resolved Billie Rinaldi
          28.
          		 Apply SLIDER-1199 to yarn native services for blacklisting nodes Sub-task Resolved Billie Rinaldi
          29.
          		 Handle InvalidResourceRequestException in native services AM onError Sub-task Resolved Billie Rinaldi
          30.
          		 Move yarn-native-service API records into hadoop-yarn-slider-core module Sub-task Resolved Jian He
          31.
          		 Default lifetime for native services app is invalid Sub-task Resolved Billie Rinaldi
          32.
          		 Refactor yarn-native-services framework Sub-task Resolved Jian He
          33.
          		 Port slider's groovy unit tests to yarn native services Sub-task Resolved Billie Rinaldi
          34.
          		 Support service assembly Sub-task Resolved Unassigned
          35.
          		 Integrate service app master to write data into ATSv2. Sub-task Resolved Rohith Sharma K S
          36.
          		 Support to add native-service specific details in new YARN UI Sub-task Resolved Akhil PB
          37.
          		 Remove some unneeded code after YARN-6255 Sub-task Resolved Jian He
          38.
          		 Improve configuring services through REST API Sub-task Resolved Jian He
          39.
          		 Revisit ATSv2 integration to ensure all required information is published Sub-task Resolved Rohith Sharma K S
          40.
          		 Support to launch new native-service from new YARN UI Sub-task Resolved Akhil PB
          41.
          		 Race condition in writing service record to registry in yarn native services Sub-task Resolved Billie Rinaldi
          42.
          		 Followup fix for YARN-6405 Sub-task Resolved Jian He
          43.
          		 Add native-service AM log4j properties in classpath Sub-task Resolved Jian He
          44.
          		 Update json validation for new native services providers Sub-task Resolved Billie Rinaldi
          45.
          		 Services API delete call first attempt usually fails Sub-task Resolved Jian He
          46.
          		 We should validate yarn native services application submission side to ensure that the hostname should be less than 63 characters Sub-task Resolved Billie Rinaldi
          47.
          		 SliderAppMaster gets an NPE when creating UnregisterComponentInstance Sub-task Resolved Unassigned
          48.
          		 Native services support for specifying component start order Sub-task Resolved Billie Rinaldi
          49.
          		 Bumping up pom file hadoop version Sub-task Resolved Panagiotis Garefalakis
          50.
          		 Allow custom hostname for docker containers in native services Sub-task Resolved Billie Rinaldi
          51.
          		 Yarn-native-service framework core rewrite Sub-task Resolved Jian He
          52.
          		 Add more UTs in yarn-native-services Sub-task Resolved Jian He
          53.
          		 Post cleanup after YARN-6903, removal of org.apache.slider package Sub-task Resolved Jian He
          54.
          		 Rename application to service in yarn-native-services Sub-task Resolved Jian He
          55.
          		 Clean up packaging and dependencies for yarn-native-services Sub-task Resolved Billie Rinaldi
          56.
          		 Create introductory site documentation for YARN native services Sub-task Resolved Gour Saha
          57.
          		 Merge yarn-native-service branch into trunk Sub-task Resolved Jian He
          58.
          		 Create install doc for YARN native services Sub-task Resolved Jian He
          59.
          		 Miscellaneous fixes in yarn-native-services Sub-task Resolved Jian He
          60.
          		 Improve yarn-service documentation Sub-task Resolved Jian He
          61.
          		 Add jsvc support for RegistryDNS Sub-task Resolved Billie Rinaldi
          62.
          		 Fix high CPU usage issue in RegistryDNS Sub-task Resolved Jian He

          Activity

            People

              Unassigned Unassigned
              vinodkv Vinod Kumar Vavilapalli
              Votes:
              0 Vote for this issue
              Watchers:
              52 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: