Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-913

Umbrella: Add a way to register long-lived services in a YARN cluster

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.5.0, 2.4.1
    • Fix Version/s: None
    • Component/s: api, resourcemanager
    • Labels:
      None

      Description

      In a YARN cluster you can't predict where services will come up -or on what ports. The services need to work those things out as they come up and then publish them somewhere.

      Applications need to be able to find the service instance they are to bond to -and not any others in the cluster.

      Some kind of service registry -in the RM, in ZK, could do this. If the RM held the write access to the ZK nodes, it would be more secure than having apps register with ZK themselves.

        Attachments

        1. 2014-09-03_Proposed_YARN_Service_Registry.pdf
          337 kB
          Steve Loughran
        2. 2014-09-08_YARN_Service_Registry.pdf
          327 kB
          Steve Loughran
        3. RegistrationServiceDetails.txt
          2 kB
          Robert Joseph Evans
        4. YARN-913-001.patch
          215 kB
          Steve Loughran
        5. YARN-913-002.patch
          261 kB
          Steve Loughran
        6. YARN-913-003.patch
          374 kB
          Steve Loughran
        7. YARN-913-003.patch
          343 kB
          Steve Loughran
        8. YARN-913-004.patch
          375 kB
          Steve Loughran
        9. YARN-913-006.patch
          389 kB
          Steve Loughran
        10. YARN-913-007.patch
          394 kB
          Steve Loughran
        11. YARN-913-008.patch
          424 kB
          Steve Loughran
        12. YARN-913-009.patch
          492 kB
          Steve Loughran
        13. YARN-913-010.patch
          497 kB
          Steve Loughran
        14. YARN-913-011.patch
          502 kB
          Steve Loughran
        15. YARN-913-012.patch
          523 kB
          Steve Loughran
        16. YARN-913-013.patch
          526 kB
          Steve Loughran
        17. YARN-913-014.patch
          532 kB
          Steve Loughran
        18. YARN-913-015.patch
          529 kB
          Steve Loughran
        19. YARN-913-016.patch
          544 kB
          Steve Loughran
        20. YARN-913-017.patch
          544 kB
          Steve Loughran
        21. YARN-913-018.patch
          545 kB
          Steve Loughran
        22. YARN-913-019.patch
          550 kB
          Steve Loughran
        23. YARN-913-020.patch
          550 kB
          Steve Loughran
        24. YARN-913-021.patch
          549 kB
          Steve Loughran
        25. yarnregistry.pdf
          168 kB
          Steve Loughran
        26. yarnregistry.pdf
          176 kB
          Steve Loughran
        27. yarnregistry.pdf
          176 kB
          Steve Loughran
        28. yarnregistry.tla
          14 kB
          Steve Loughran

        Issue Links

        1.
        RM to support YARN registry Sub-task Resolved Steve Loughran   Actions
        2.
        Add CLI client to the registry to list, view and manipulate entries Sub-task Closed Akshay Radia   Actions
        3.
        distributed shell & tests to use registry Sub-task Open Steve Loughran   Actions
        4.
        add hadoop-yarn-registry package under hadoop-yarn Sub-task Closed Steve Loughran   Actions
        5.
        MiniYARNCluster to (optionally) add MicroZookeeper service Sub-task Resolved Steve Loughran   Actions
        6.
        yarn-registry JAR won't link against ZK 3.4.5 Sub-task Closed Steve Loughran

        0%

        Original Estimate - 0.5h
        Remaining Estimate - 0.5h
        Actions
        7.
        registry punycoding of usernames doesn't fix all usernames to be DNS-valid Sub-task Closed Steve Loughran   Actions
        8.
        Improved Yarn Registry service record structure Sub-task Closed Steve Loughran   Actions
        9.
        registry config options: document and move to core-default Sub-task Closed Steve Loughran

        66%

        Original Estimate - 1h Original Estimate - 1h
        Time Spent - 1h Remaining Estimate - 0.5h
        Actions
        10.
        TestSecureRMRegistryOperations failing on windows: secure ZK won't start Sub-task Closed Steve Loughran   Actions
        11.
        ktutil test hanging on some machines/ktutil versions Sub-task Closed Steve Loughran   Actions
        12.
        TestSecureRMRegistryOperations failing on windows: auth problems Sub-task Closed Steve Loughran   Actions
        13.
        Provide Read Write REST view of YARN registry and client compatible with existing Registry API Sub-task Open Unassigned   Actions
        14.
        TTL for YARN Registry SRV records Sub-task Open Unassigned   Actions
        15.
        Collision-free unique bindings & refresh APIs for service records Sub-task Open Unassigned   Actions
        16.
        TTL & identity aware read cache for the SRV records Sub-task Open Unassigned   Actions
        17.
        clean up registry code for java 7/8 Sub-task Resolved Steve Loughran   Actions
        18.
        YARN registry service should avoid scanning whole ZK tree for every container/application finish Sub-task Resolved Wangda Tan   Actions

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              stevel@apache.org Steve Loughran Assign to me
              Reporter:
              stevel@apache.org Steve Loughran

              Dates

              • Created:
                Updated:

              Time Tracking

              Estimated:
              Original Estimate - 1.5h Original Estimate - 1.5h
              1.5h
              Remaining:
              Time Spent - 1h Remaining Estimate - 1h
              1h
              Logged:
              Time Spent - 1h Remaining Estimate - 1h
              1h

                Issue deployment