Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5079

[Umbrella] Native YARN framework layer for services and beyond

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersStop watchingWatchersCreate sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Release Note:
      A framework is implemented to orchestrate containers on YARN

      Description

      (See overview doc at YARN-4692, modifying and copy-pasting some of the relevant pieces and sub-section 3.3.1 to track the specific sub-item.)

      (This is a companion to YARN-4793 in our effort to simplify the entire story, but focusing on APIs)

      So far, YARN by design has restricted itself to having a very low-­level API that can support any type of application. Frameworks like Apache Hadoop MapReduce, Apache Tez, Apache Spark, Apache REEF, Apache Twill, Apache Helix and others ended up exposing higher level APIs that end­-users can directly leverage to build their applications on top of YARN. On the services side, Apache Slider has done something similar.

      With our current attention on making services first­-class and simplified, it's time to take a fresh look at how we can make Apache Hadoop YARN support services well out of the box. Beyond the functionality that I outlined in the previous sections in the doc on how NodeManagers can be enhanced to help services, the biggest missing piece is the framework itself. There is a lot of very important functionality that a services' framework can own together with YARN in executing services end­-to­-end.

      In this JIRA I propose we look at having a native Apache Hadoop framework for running services natively on YARN.

        Attachments

        Issue Links

        1.
        Port initial slider-core module code into yarn Sub-task Resolved Jian He Actions
        2.
        Create an agent-less docker provider in the native-services framework Sub-task Resolved Billie Rinaldi Actions
        3.
        Move Java only tests from slider develop to yarn-native-services Sub-task Resolved Gour Saha Actions
        4.
        Apply SLIDER-875 to yarn-native-services for supporting a composite app-of-apps Sub-task Resolved Billie Rinaldi Actions
        5.
        Apply SLIDER-1166 to yarn-native-services branch Sub-task Resolved Gour Saha Actions
        6.
        Add 2 new fields in Slider status output - image-name and is-privileged-container Sub-task Resolved Billie Rinaldi Actions
        7.
        Integrate native services modules into maven build Sub-task Resolved Billie Rinaldi Actions
        8.
        Fix issues in yarn native services apps-of-apps Sub-task Resolved Billie Rinaldi Actions
        9.
        Add a new field in Slider status output - lifetime (remaining) Sub-task Resolved Jian He Actions
        10.
        Add .keep file for yarn native services AM web app Sub-task Resolved Billie Rinaldi Actions
        11.
        Slider version output includes variables buildNumber and hadoop.version Sub-task Resolved Gour Saha Actions
        12.
        Native services client errors out when config formats are uppercase Sub-task Resolved Billie Rinaldi Actions
        13.
        Remove the agent code in slider Sub-task Resolved Jian He Actions
        14.
        Slider handles "per.component" for multiple components incorrectly Sub-task Resolved Billie Rinaldi Actions
        15.
        Write native services container stderr file to log directory Sub-task Resolved Billie Rinaldi Actions
        16.
        Native services AM should remain up if RM is down Sub-task Resolved Billie Rinaldi Actions
        17.
        Add Slider debug config in yarn log4j Sub-task Resolved Unassigned Actions
        18.
        Fix ASF license warnings for slider core module Sub-task Resolved Billie Rinaldi Actions
        19.
        Generate native services protobuf classes during build Sub-task Resolved Billie Rinaldi Actions
        20.
        Fix slider core module findbugs warnings Sub-task Resolved Jian He Actions
        21.
        Fix slider core module javadocs Sub-task Resolved Billie Rinaldi Actions
        22.
        Remove the agent - slider AM ssl related code Sub-task Resolved Jian He Actions
        23.
        Allow native services quicklinks to be exported for each component Sub-task Resolved Billie Rinaldi Actions
        24.
        Native services AM kills app on AMRMClientAsync onError call Sub-task Resolved Billie Rinaldi Actions
        25.
        Native services AM kills app on AMRMClientAsync onError call Sub-task Resolved Billie Rinaldi Actions
        26.
        Followup fix for slider core module findbugs Sub-task Resolved Jian He Actions
        27.
        Create an agent-less docker-less provider in the native services framework Sub-task Resolved Billie Rinaldi Actions
        28.
        Apply SLIDER-1199 to yarn native services for blacklisting nodes Sub-task Resolved Billie Rinaldi Actions
        29.
        Handle InvalidResourceRequestException in native services AM onError Sub-task Resolved Billie Rinaldi Actions
        30.
        Move yarn-native-service API records into hadoop-yarn-slider-core module Sub-task Resolved Jian He Actions
        31.
        Default lifetime for native services app is invalid Sub-task Resolved Billie Rinaldi Actions
        32.
        Refactor yarn-native-services framework Sub-task Resolved Jian He Actions
        33.
        Port slider's groovy unit tests to yarn native services Sub-task Resolved Billie Rinaldi Actions
        34.
        Support service assembly Sub-task Resolved Unassigned Actions
        35.
        Integrate service app master to write data into ATSv2. Sub-task Resolved Rohith Sharma K S Actions
        36.
        Support to add native-service specific details in new YARN UI Sub-task Resolved Akhil PB Actions
        37.
        Remove some unneeded code after YARN-6255 Sub-task Resolved Jian He Actions
        38.
        Improve configuring services through REST API Sub-task Resolved Jian He Actions
        39.
        Revisit ATSv2 integration to ensure all required information is published Sub-task Resolved Rohith Sharma K S Actions
        40.
        Support to launch new native-service from new YARN UI Sub-task Resolved Akhil PB Actions
        41.
        Race condition in writing service record to registry in yarn native services Sub-task Resolved Billie Rinaldi Actions
        42.
        Followup fix for YARN-6405 Sub-task Resolved Jian He Actions
        43.
        Add native-service AM log4j properties in classpath Sub-task Resolved Jian He Actions
        44.
        Update json validation for new native services providers Sub-task Resolved Billie Rinaldi Actions
        45.
        Services API delete call first attempt usually fails Sub-task Resolved Jian He Actions
        46.
        We should validate yarn native services application submission side to ensure that the hostname should be less than 63 characters Sub-task Resolved Billie Rinaldi Actions
        47.
        SliderAppMaster gets an NPE when creating UnregisterComponentInstance Sub-task Resolved Unassigned Actions
        48.
        Native services support for specifying component start order Sub-task Resolved Billie Rinaldi Actions
        49.
        Bumping up pom file hadoop version Sub-task Resolved Panagiotis Garefalakis Actions
        50.
        Allow custom hostname for docker containers in native services Sub-task Resolved Billie Rinaldi Actions
        51.
        Yarn-native-service framework core rewrite Sub-task Resolved Jian He Actions
        52.
        Add more UTs in yarn-native-services Sub-task Resolved Jian He Actions
        53.
        Post cleanup after YARN-6903, removal of org.apache.slider package Sub-task Resolved Jian He Actions
        54.
        Rename application to service in yarn-native-services Sub-task Resolved Jian He Actions
        55.
        Clean up packaging and dependencies for yarn-native-services Sub-task Resolved Billie Rinaldi Actions
        56.
        Create introductory site documentation for YARN native services Sub-task Resolved Gour Saha Actions
        57.
        Merge yarn-native-service branch into trunk Sub-task Resolved Jian He Actions
        58.
        Create install doc for YARN native services Sub-task Resolved Jian He Actions
        59.
        Miscellaneous fixes in yarn-native-services Sub-task Resolved Jian He Actions
        60.
        Improve yarn-service documentation Sub-task Resolved Jian He Actions
        61.
        Add jsvc support for RegistryDNS Sub-task Resolved Billie Rinaldi Actions
        62.
        Fix high CPU usage issue in RegistryDNS Sub-task Resolved Jian He Actions

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              Unassigned Assign to me
              Reporter:
              vinodkv Vinod Kumar Vavilapalli

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment