Uploaded image for project: 'Slider'
  1. Slider
  2. SLIDER-82

Support ANTI_AFFINITY_REQUIRED option

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Slider 0.91
    • Component/s: appmaster
    • Labels:
      None
    • Sprint:
      Slider September #1

      Description

      slider has an anti-affinity flag in roles (visible in resources.json?), which is ignored.

      YARN-1042 promises this for YARN, slider will need

      1. flag in resources.json
      2. use in container requests

      we may also want two policies: anti-affinity-desired, and -required. Then if required nodes get >1 container for the same component type on the same node, it'd have to request a new one and return the old one (Risk: getting the same one back).

        Attachments

          Issue Links

          1.
          build node map from yarn update reports; serve via REST/IPC Sub-task Resolved Steve Loughran

          0%

          Original Estimate - 2h
          Remaining Estimate - 2h
          2.
          Write mock/unit tests for AA placement Sub-task Resolved Steve Loughran  
          3.
          RoleHistory to (carefully) share RoleStatus instances with AppState Sub-task Resolved Steve Loughran  
          4.
          Implement sequential AA assignment without location constraints Sub-task Resolved Steve Loughran  
          5.
          Use nodemap to build up location restrictions on AA placement Sub-task Resolved Steve Loughran

          0%

          Original Estimate - 8h
          Remaining Estimate - 8h
          6.
          Add mock test to simulate AA failure and restart of AA request sequence Sub-task Resolved Steve Loughran

          100%

          Original Estimate - 2h
          Time Spent - 0.5h Time Not Required
          7.
          Add functional tests of AA placement Sub-task Resolved Steve Loughran  
          8.
          revisit why OutstandingRequest.buildContainerRequest() sets label==null on a placed request Sub-task Resolved Unassigned  
          9.
          AM web UI to show state of AA request Sub-task Resolved Steve Loughran  
          10.
          add minicluster test of AA placement Sub-task Resolved Steve Loughran

          0%

          Original Estimate - 2h
          Remaining Estimate - 2h
          11.
          add mock test of failure of AA container and re-request; fix any failures Sub-task Resolved Steve Loughran

          0%

          Original Estimate - 1h
          Remaining Estimate - 1h
          12.
          Mock AA test for nodemap not updated Sub-task Resolved Steve Loughran  
          13.
          add "nodemap" command to get the (JSON) nodemap of the YARN cluster Sub-task Resolved Steve Loughran  
          14.
          Update slider docs with coverage of AA placement Sub-task Resolved Steve Loughran

          0%

          Original Estimate - 3h
          Remaining Estimate - 3h
          15.
          AM can't make YarnClient calls on a secure cluster Sub-task Resolved Steve Loughran  
          16.
          review label logic in AA code Sub-task Resolved Steve Loughran  
          17.
          add regression test to verify latest role history format reload Sub-task Resolved Steve Loughran  
          18.
          rename NO_DATA_LOCALITY placement policy to ANYWHERE Sub-task Resolved Steve Loughran  

            Activity

              People

              • Assignee:
                stevel@apache.org Steve Loughran
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                1 Vote for this issue
                Watchers:
                21 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 186h
                  186h
                  Remaining:
                  Remaining Estimate - 184h
                  184h
                  Logged:
                  Remaining Estimate - 184h Time Not Required
                  0.5h