Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4793

[Umbrella] Simplified API layer for services and beyond

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.1.0
    • Fix Version/s: 3.1.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Release Note:
      A REST API service is implemented to enable users to launch and manage container based services on YARN via REST API

      Description

      [See overview doc at YARN-4692, modifying and copy-pasting some of the relevant pieces and sub-section 3.3.2 to track the specific sub-item.]

      Bringing a new service on YARN today is not a simple experience. The APIs of existing frameworks are either too low­ level (native YARN), require writing new code (for frameworks with programmatic APIs ) or writing a complex spec (for declarative frameworks).

      In addition to building critical building blocks inside YARN (as part of other efforts at YARN-4692), we should also look to simplifying the user facing story for building services. Experience of projects like Slider building real-­life services like HBase, Storm, Accumulo, Solr etc gives us some very good learnings on how simplified APIs for building services will look like.

      To this end, we should look at a new simple-services API layer backed by REST interfaces. The REST layer can act as a single point of entry for creation and lifecycle management of YARN services. Services here can range from simple single-­component apps to the most complex, multi­-component applications needing special orchestration needs.

      We should also look at making this a unified REST based entry point for other important features like resource­-profile management (YARN-3926), package-definitions' lifecycle­-management and service­-discovery (YARN-913 / YARN-4757). We also need to flesh out its relation to our present much ­lower level REST APIs (YARN-1695) in YARN for application-­submission and management.

        Attachments

          Issue Links

          1.
          Initial code for native services REST API Sub-task Resolved Gour Saha
          2.
          Checkin swagger definition in the repo Sub-task Resolved Gour Saha
          3.
          Update native services REST API to use agentless docker provider Sub-task Resolved Gour Saha
          4.
          Bug fixes identified during testing Sub-task Resolved Gour Saha
          5.
          Make the service REST API use the app timeout feature YARN-4205 Sub-task Resolved Jian He
          6.
          Integrate remaining app lifetime using feature implemented in YARN-4206 Sub-task Resolved Jian He
          7.
          Integrate update app lifetime using feature implemented in YARN-5611 Sub-task Resolved Jian He
          8.
          Performance improvement of native-services REST API service Sub-task Resolved Gour Saha
          9.
          Convert enums in swagger definition to uppercase Sub-task Resolved Gour Saha
          10.
          Convert enums values in service code to upper case and special handling of an error Sub-task Resolved Gour Saha
          11.
          Add gc log options to the yarn daemon script when starting services-api Sub-task Resolved Billie Rinaldi
          12.
          Exception during GET call - "Failed to retrieve application: null" Sub-task Resolved Gour Saha
          13.
          Slider should not try to set a negative lifetime timeout value Sub-task Resolved Jian He
          14.
          DELETE call sometimes returns success when app is not deleted Sub-task Resolved Gour Saha
          15.
          Correctly handle docker.image and launch command when unique_component_support is specified Sub-task Resolved Unassigned
          16.
          Avoid or eliminate expensive YARN get all applications call Sub-task Resolved Gour Saha
          17.
          Fix native services API JsonMappingException for PlacementPolicy Sub-task Resolved Gour Saha
          18.
          Fix findbugs, site warnings in yarn-services-api module Sub-task Resolved Jian He
          19.
          Few additional paths in Slider client still uses get all Applications without tags/states filter Sub-task Resolved Gour Saha
          20.
          SliderClient bondToCluster should call findInstance with live states filter Sub-task Resolved Gour Saha
          21.
          Add artifact info and privileged container details to the container info in API GET response Sub-task Resolved Gour Saha
          22.
          Rename the API Application to Service Sub-task Resolved Jian He
          23.
          Rest API site documentation Sub-task Resolved Gour Saha
          24.
          YARN service api can not reuse json file serialized in hdfs Sub-task Resolved Eric Yang

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                vinodkv Vinod Kumar Vavilapalli
              • Votes:
                0 Vote for this issue
                Watchers:
                33 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: