Hadoop YARN
  1. Hadoop YARN
  2. YARN-913

Umbrella: Add a way to register long-lived services in a YARN cluster

    Details

    • Type: New Feature New Feature
    • Status: In Progress
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 2.5.0, 2.4.1
    • Fix Version/s: None
    • Component/s: api, resourcemanager
    • Labels:
      None
    • Target Version/s:

      Description

      In a YARN cluster you can't predict where services will come up -or on what ports. The services need to work those things out as they come up and then publish them somewhere.

      Applications need to be able to find the service instance they are to bond to -and not any others in the cluster.

      Some kind of service registry -in the RM, in ZK, could do this. If the RM held the write access to the ZK nodes, it would be more secure than having apps register with ZK themselves.

      1. 2014-09-03_Proposed_YARN_Service_Registry.pdf
        337 kB
        Steve Loughran
      2. 2014-09-08_YARN_Service_Registry.pdf
        327 kB
        Steve Loughran
      3. RegistrationServiceDetails.txt
        2 kB
        Robert Joseph Evans
      4. YARN-913-001.patch
        215 kB
        Steve Loughran
      5. YARN-913-002.patch
        261 kB
        Steve Loughran
      6. YARN-913-003.patch
        374 kB
        Steve Loughran
      7. YARN-913-003.patch
        343 kB
        Steve Loughran
      8. YARN-913-004.patch
        375 kB
        Steve Loughran
      9. YARN-913-006.patch
        389 kB
        Steve Loughran
      10. YARN-913-007.patch
        394 kB
        Steve Loughran
      11. YARN-913-008.patch
        424 kB
        Steve Loughran
      12. YARN-913-009.patch
        492 kB
        Steve Loughran
      13. YARN-913-010.patch
        497 kB
        Steve Loughran
      14. YARN-913-011.patch
        502 kB
        Steve Loughran
      15. YARN-913-012.patch
        523 kB
        Steve Loughran
      16. YARN-913-013.patch
        526 kB
        Steve Loughran
      17. YARN-913-014.patch
        532 kB
        Steve Loughran
      18. YARN-913-015.patch
        529 kB
        Steve Loughran
      19. YARN-913-016.patch
        544 kB
        Steve Loughran
      20. YARN-913-017.patch
        544 kB
        Steve Loughran
      21. YARN-913-018.patch
        545 kB
        Steve Loughran
      22. YARN-913-019.patch
        550 kB
        Steve Loughran
      23. YARN-913-020.patch
        550 kB
        Steve Loughran
      24. YARN-913-021.patch
        549 kB
        Steve Loughran
      25. yarnregistry.pdf
        168 kB
        Steve Loughran
      26. yarnregistry.pdf
        176 kB
        Steve Loughran
      27. yarnregistry.pdf
        176 kB
        Steve Loughran
      28. yarnregistry.tla
        14 kB
        Steve Loughran

        Issue Links

          Activity

          Junping Du made changes -
          Target Version/s 2.6.0 [ 12327197 ] 2.8.0 [ 12329059 ]
          Steve Loughran made changes -
          Link This issue depends upon HADOOP-11102 [ HADOOP-11102 ]
          Steve Loughran made changes -
          Link This issue depends upon HADOOP-11102 [ HADOOP-11102 ]
          Steve Loughran made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Summary Add a way to register long-lived services in a YARN cluster Umbrella: Add a way to register long-lived services in a YARN cluster
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment YARN-913-021.patch [ 12673478 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment YARN-913-020.patch [ 12673460 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment YARN-913-019.patch [ 12673424 ]
          Steve Loughran made changes -
          Attachment yarnregistry.pdf [ 12673420 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment YARN-913-018.patch [ 12673044 ]
          Steve Loughran made changes -
          Attachment yarnregistry.pdf [ 12673043 ]
          Steve Loughran made changes -
          Attachment YARN-913-017.patch [ 12673042 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment YARN-913-016.patch [ 12672406 ]
          Steve Loughran made changes -
          Attachment YARN-913-015.patch [ 12672349 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment YARN-913-014.patch [ 12672203 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment YARN-913-013.patch [ 12671701 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment YARN-913-012.patch [ 12671529 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment YARN-913-011.patch [ 12671457 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment YARN-913-010.patch [ 12671298 ]
          Steve Loughran made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment YARN-913-009.patch [ 12671012 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Link This issue is related to HADOOP-11117 [ HADOOP-11117 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment YARN-913-008.patch [ 12670048 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment YARN-913-007.patch [ 12670013 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment YARN-913-006.patch [ 12669983 ]
          Steve Loughran made changes -
          Link This issue is related to HADOOP-11111 [ HADOOP-11111 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment YARN-913-004.patch [ 12669814 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment YARN-913-003.patch [ 12669798 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment YARN-913-003.patch [ 12669504 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment YARN-913-002.patch [ 12667181 ]
          Steve Loughran made changes -
          Attachment 2014-09-08_YARN_Service_Registry.pdf [ 12667177 ]
          Attachment yarnregistry.pdf [ 12667178 ]
          Attachment yarnregistry.tla [ 12667179 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment YARN-913-001.patch [ 12666278 ]
          Steve Loughran made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Steve Loughran made changes -
          Steve Loughran made changes -
          Assignee Robert Joseph Evans [ revans2 ] Steve Loughran [ stevel@apache.org ]
          Affects Version/s 2.4.1 [ 12326697 ]
          Affects Version/s 2.5.0 [ 12326262 ]
          Affects Version/s 3.0.0 [ 12323268 ]
          Target Version/s 2.6.0 [ 12327197 ]
          Component/s resourcemanager [ 12319322 ]
          Steve Loughran made changes -
          Link This issue is depended upon by SLIDER-365 [ SLIDER-365 ]
          Steve Loughran made changes -
          Link This issue is depended upon by SLIDER-149 [ SLIDER-149 ]
          Robert Joseph Evans made changes -
          Attachment RegistrationServiceDetails.txt [ 12607606 ]
          Robert Joseph Evans made changes -
          Assignee Robert Joseph Evans [ revans2 ]
          Steve Loughran made changes -
          Link This issue relates to YARN-1143 [ YARN-1143 ]
          Steve Loughran made changes -
          Field Original Value New Value
          Link This issue is depended upon by YARN-896 [ YARN-896 ]
          Steve Loughran created issue -

            People

            • Assignee:
              Steve Loughran
              Reporter:
              Steve Loughran
            • Votes:
              4 Vote for this issue
              Watchers:
              59 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - 1.5h Original Estimate - 1.5h
                1.5h
                Remaining:
                Time Spent - 1h Remaining Estimate - 1h
                1h
                Logged:
                Time Spent - 1h Remaining Estimate - 1h
                1h

                  Development