Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-913

Umbrella: Add a way to register long-lived services in a YARN cluster

    Details

    • Type: New Feature
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.5.0, 2.4.1
    • Fix Version/s: None
    • Component/s: api, resourcemanager
    • Labels:
      None

      Description

      In a YARN cluster you can't predict where services will come up -or on what ports. The services need to work those things out as they come up and then publish them somewhere.

      Applications need to be able to find the service instance they are to bond to -and not any others in the cluster.

      Some kind of service registry -in the RM, in ZK, could do this. If the RM held the write access to the ZK nodes, it would be more secure than having apps register with ZK themselves.

        Attachments

        1. 2014-09-03_Proposed_YARN_Service_Registry.pdf
          337 kB
          Steve Loughran
        2. 2014-09-08_YARN_Service_Registry.pdf
          327 kB
          Steve Loughran
        3. RegistrationServiceDetails.txt
          2 kB
          Robert Joseph Evans
        4. YARN-913-001.patch
          215 kB
          Steve Loughran
        5. YARN-913-002.patch
          261 kB
          Steve Loughran
        6. YARN-913-003.patch
          374 kB
          Steve Loughran
        7. YARN-913-003.patch
          343 kB
          Steve Loughran
        8. YARN-913-004.patch
          375 kB
          Steve Loughran
        9. YARN-913-006.patch
          389 kB
          Steve Loughran
        10. YARN-913-007.patch
          394 kB
          Steve Loughran
        11. YARN-913-008.patch
          424 kB
          Steve Loughran
        12. YARN-913-009.patch
          492 kB
          Steve Loughran
        13. YARN-913-010.patch
          497 kB
          Steve Loughran
        14. YARN-913-011.patch
          502 kB
          Steve Loughran
        15. YARN-913-012.patch
          523 kB
          Steve Loughran
        16. YARN-913-013.patch
          526 kB
          Steve Loughran
        17. YARN-913-014.patch
          532 kB
          Steve Loughran
        18. YARN-913-015.patch
          529 kB
          Steve Loughran
        19. YARN-913-016.patch
          544 kB
          Steve Loughran
        20. YARN-913-017.patch
          544 kB
          Steve Loughran
        21. YARN-913-018.patch
          545 kB
          Steve Loughran
        22. YARN-913-019.patch
          550 kB
          Steve Loughran
        23. YARN-913-020.patch
          550 kB
          Steve Loughran
        24. YARN-913-021.patch
          549 kB
          Steve Loughran
        25. yarnregistry.pdf
          168 kB
          Steve Loughran
        26. yarnregistry.pdf
          176 kB
          Steve Loughran
        27. yarnregistry.pdf
          176 kB
          Steve Loughran
        28. yarnregistry.tla
          14 kB
          Steve Loughran

          Issue Links

          1.
          RM to support YARN registry Sub-task Resolved Steve Loughran  
          2.
          Add CLI client to the registry to list, view and manipulate entries Sub-task Closed Akshay Radia  
          3.
          distributed shell & tests to use registry Sub-task Open Steve Loughran  
          4.
          add hadoop-yarn-registry package under hadoop-yarn Sub-task Closed Steve Loughran  
          5.
          MiniYARNCluster to (optionally) add MicroZookeeper service Sub-task Open Steve Loughran  
          6.
          yarn-registry JAR won't link against ZK 3.4.5 Sub-task Closed Steve Loughran

          0%

          Original Estimate - 0.5h
          Remaining Estimate - 0.5h
          7.
          registry punycoding of usernames doesn't fix all usernames to be DNS-valid Sub-task Closed Steve Loughran  
          8.
          Improved Yarn Registry service record structure Sub-task Closed Steve Loughran  
          9.
          registry config options: document and move to core-default Sub-task Closed Steve Loughran

          66%

          Original Estimate - 1h Original Estimate - 1h
          Time Spent - 1h Remaining Estimate - 0.5h
          10.
          TestSecureRMRegistryOperations failing on windows: secure ZK won't start Sub-task Closed Steve Loughran  
          11.
          ktutil test hanging on some machines/ktutil versions Sub-task Closed Steve Loughran  
          12.
          TestSecureRMRegistryOperations failing on windows: auth problems Sub-task Closed Steve Loughran  
          13.
          Provide Read Write REST view of YARN registry and client compatible with existing Registry API Sub-task Open Unassigned  
          14.
          TTL for YARN Registry SRV records Sub-task Open Unassigned  
          15.
          Collision-free unique bindings & refresh APIs for service records Sub-task Open Unassigned  
          16.
          TTL & identity aware read cache for the SRV records Sub-task Open Unassigned  
          17.
          clean up registry code for java 7/8 Sub-task Patch Available Steve Loughran  
          18.
          YARN registry service should avoid scanning whole ZK tree for every container/application finish Sub-task Resolved Wangda Tan  

            Activity

              People

              • Assignee:
                stevel@apache.org Steve Loughran
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                5 Vote for this issue
                Watchers:
                69 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - 1.5h Original Estimate - 1.5h
                  1.5h
                  Remaining:
                  Time Spent - 1h Remaining Estimate - 1h
                  1h
                  Logged:
                  Time Spent - 1h Remaining Estimate - 1h
                  1h