Uploaded image for project: 'Slider'
  1. Slider
  2. SLIDER-82

Support ANTI_AFFINITY_REQUIRED option

Details

    • Task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • Slider 0.91
    • appmaster
    • None
    • Slider September #1

    Description

      slider has an anti-affinity flag in roles (visible in resources.json?), which is ignored.

      YARN-1042 promises this for YARN, slider will need

      1. flag in resources.json
      2. use in container requests

      we may also want two policies: anti-affinity-desired, and -required. Then if required nodes get >1 container for the same component type on the same node, it'd have to request a new one and return the old one (Risk: getting the same one back).

      Attachments

        1. SLIDER-82.002.patch
          14 kB
          Rajesh Kartha

        Issue Links

          1.
          build node map from yarn update reports; serve via REST/IPC Sub-task Resolved Steve Loughran

          0%

          Original Estimate - 2h
          Remaining Estimate - 2h
          Actions
          2.
          Write mock/unit tests for AA placement Sub-task Resolved Steve Loughran   Actions
          3.
          RoleHistory to (carefully) share RoleStatus instances with AppState Sub-task Resolved Steve Loughran   Actions
          4.
          Implement sequential AA assignment without location constraints Sub-task Resolved Steve Loughran   Actions
          5.
          Use nodemap to build up location restrictions on AA placement Sub-task Resolved Steve Loughran

          0%

          Original Estimate - 8h
          Remaining Estimate - 8h
          Actions
          6.
          Add mock test to simulate AA failure and restart of AA request sequence Sub-task Resolved Steve Loughran

          100%

          Original Estimate - 2h
          Time Spent - 0.5h Time Not Required
          Actions
          7.
          Add functional tests of AA placement Sub-task Resolved Steve Loughran   Actions
          8.
          revisit why OutstandingRequest.buildContainerRequest() sets label==null on a placed request Sub-task Resolved Unassigned   Actions
          9.
          AM web UI to show state of AA request Sub-task Resolved Steve Loughran   Actions
          10.
          add minicluster test of AA placement Sub-task Resolved Steve Loughran

          0%

          Original Estimate - 2h
          Remaining Estimate - 2h
          Actions
          11.
          add mock test of failure of AA container and re-request; fix any failures Sub-task Resolved Steve Loughran

          0%

          Original Estimate - 1h
          Remaining Estimate - 1h
          Actions
          12.
          Mock AA test for nodemap not updated Sub-task Resolved Steve Loughran   Actions
          13.
          add "nodemap" command to get the (JSON) nodemap of the YARN cluster Sub-task Resolved Steve Loughran   Actions
          14.
          Update slider docs with coverage of AA placement Sub-task Resolved Steve Loughran

          0%

          Original Estimate - 3h
          Remaining Estimate - 3h
          Actions
          15.
          AM can't make YarnClient calls on a secure cluster Sub-task Resolved Steve Loughran   Actions
          16.
          review label logic in AA code Sub-task Resolved Steve Loughran   Actions
          17.
          add regression test to verify latest role history format reload Sub-task Resolved Steve Loughran   Actions
          18.
          rename NO_DATA_LOCALITY placement policy to ANYWHERE Sub-task Resolved Steve Loughran   Actions

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              1 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 186h
                  186h
                  Remaining:
                  Remaining Estimate - 184h
                  184h
                  Logged:
                  Remaining Estimate - 184h Time Not Required
                  0.5h

                  Agile

                    Active Sprint:
                    Slider September #1 ends 11/Sep/15
                    View on Board

                    Slack

                      Issue deployment