Uploaded image for project: 'Slider'
  1. Slider
  2. SLIDER-82

Support ANTI_AFFINITY_REQUIRED option

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Slider 0.91
    • Component/s: appmaster
    • Labels:
      None
    • Sprint:
      Slider September #1

      Description

      slider has an anti-affinity flag in roles (visible in resources.json?), which is ignored.

      YARN-1042 promises this for YARN, slider will need

      1. flag in resources.json
      2. use in container requests

      we may also want two policies: anti-affinity-desired, and -required. Then if required nodes get >1 container for the same component type on the same node, it'd have to request a new one and return the old one (Risk: getting the same one back).

        Issue Links

        1.
        build node map from yarn update reports; serve via REST/IPC Sub-task Resolved Steve Loughran

        0%

        Original Estimate - 2h
        Remaining Estimate - 2h
         
        2.
        Write mock/unit tests for AA placement Sub-task Resolved Steve Loughran  
         
        3.
        RoleHistory to (carefully) share RoleStatus instances with AppState Sub-task Resolved Steve Loughran  
         
        4.
        Implement sequential AA assignment without location constraints Sub-task Resolved Steve Loughran  
         
        5.
        Use nodemap to build up location restrictions on AA placement Sub-task Resolved Steve Loughran

        0%

        Original Estimate - 8h
        Remaining Estimate - 8h
         
        6.
        Add mock test to simulate AA failure and restart of AA request sequence Sub-task Resolved Steve Loughran

        100%

        Original Estimate - 2h
        Time Spent - 0.5h Time Not Required
         
        7.
        Add functional tests of AA placement Sub-task Resolved Steve Loughran  
         
        8.
        revisit why OutstandingRequest.buildContainerRequest() sets label==null on a placed request Sub-task Resolved Unassigned  
         
        9.
        AM web UI to show state of AA request Sub-task Resolved Steve Loughran  
         
        10.
        add minicluster test of AA placement Sub-task Resolved Steve Loughran

        0%

        Original Estimate - 2h
        Remaining Estimate - 2h
         
        11.
        add mock test of failure of AA container and re-request; fix any failures Sub-task Resolved Steve Loughran

        0%

        Original Estimate - 1h
        Remaining Estimate - 1h
         
        12.
        Mock AA test for nodemap not updated Sub-task Resolved Steve Loughran  
         
        13.
        add "nodemap" command to get the (JSON) nodemap of the YARN cluster Sub-task Resolved Steve Loughran  
         
        14.
        Update slider docs with coverage of AA placement Sub-task Resolved Steve Loughran

        0%

        Original Estimate - 3h
        Remaining Estimate - 3h
         
        15.
        AM can't make YarnClient calls on a secure cluster Sub-task Resolved Steve Loughran  
         
        16.
        review label logic in AA code Sub-task Resolved Steve Loughran  
         
        17.
        add regression test to verify latest role history format reload Sub-task Resolved Steve Loughran  
         
        18.
        rename NO_DATA_LOCALITY placement policy to ANYWHERE Sub-task Resolved Steve Loughran  
         

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          SLIDER-823 contains a strategy we could use to implement anti-affinity based on the SLIDER-799 escalation code

          Show
          stevel@apache.org Steve Loughran added a comment - SLIDER-823 contains a strategy we could use to implement anti-affinity based on the SLIDER-799 escalation code
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I'm thinking we can do it in Slider. This lets us use node failure history in the placement requests, allow us to decide if/when to permit same-host deployment. Doing it in slider will allow us to do this to run on existing "ODP-compliant " hadoop 2.6 clusters, rather than waiting for a YARN feature and requiring cluster upgrades.

          We will have to read in topology information; this includes picking up any information on failure domains to try and spread load away from hosts that share same failure risks (e.g. multiple servers in a 1U/2U unit sharing power and network)

          Show
          stevel@apache.org Steve Loughran added a comment - I'm thinking we can do it in Slider. This lets us use node failure history in the placement requests, allow us to decide if/when to permit same-host deployment. Doing it in slider will allow us to do this to run on existing "ODP-compliant " hadoop 2.6 clusters, rather than waiting for a YARN feature and requiring cluster upgrades. We will have to read in topology information; this includes picking up any information on failure domains to try and spread load away from hosts that share same failure risks (e.g. multiple servers in a 1U/2U unit sharing power and network)
          Hide
          yanghaogn Yang Hao added a comment -

          It's like TWILL-87

          Show
          yanghaogn Yang Hao added a comment - It's like TWILL-87
          Hide
          stevel@apache.org Steve Loughran added a comment -

          well spotted. I knew they were talking about it, didn't know it was up. They also do placement escalation and timeout; don't know about node reliability tracking.

          Show
          stevel@apache.org Steve Loughran added a comment - well spotted. I knew they were talking about it, didn't know it was up. They also do placement escalation and timeout; don't know about node reliability tracking.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          As I commented on YARN-1042, the YARN effort is complex and will likely take a while. This JIRA could take a shortcut by implementing it in the AM - it won't be fool-proof without RM support, but it should get you off the ground.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - As I commented on YARN-1042 , the YARN effort is complex and will likely take a while. This JIRA could take a shortcut by implementing it in the AM - it won't be fool-proof without RM support, but it should get you off the ground.
          Hide
          kartha Rajesh Kartha added a comment -

          Based on the discussion on the Slider dev list:
          https://www.mail-archive.com/search?l=dev@slider.incubator.apache.org&q=subject:%22Re%3A+A+basic+approach+towards+implementing+the+ANTI-AFFINITY+in+Slider%22&o=newest&f=1

          here is an early patch that tries to implement ANTI_AFFINITY_REQUIRED using:

          1) The updateBlacklist() method from AMRMClientAsyncImpl introduced in YARN-1723, which can blacklist the nodes on which the component with ANTI_AFFINITY_REQUIRED is already running

          2) Additional check for containers to be on already requested nodes and discarding them during restart

          Needless to say, this patch requires Hadoop 2.7.0 to build.

          My testing with the Slider-HBase package using REGION_SERVER with ANTI_AFFINITY_REQUIRED flag does seem to function as expected with all the checks added, getting enforced during flex. Also any containers that got allocated on already requested nodes during restart were being discarded.

          Will investigate and add some unit tests soon.

          Meanwhile, it will be useful, if someone can review the patch and add their thought/comments.

          Show
          kartha Rajesh Kartha added a comment - Based on the discussion on the Slider dev list: https://www.mail-archive.com/search?l=dev@slider.incubator.apache.org&q=subject:%22Re%3A+A+basic+approach+towards+implementing+the+ANTI-AFFINITY+in+Slider%22&o=newest&f=1 here is an early patch that tries to implement ANTI_AFFINITY_REQUIRED using: 1) The updateBlacklist() method from AMRMClientAsyncImpl introduced in YARN-1723 , which can blacklist the nodes on which the component with ANTI_AFFINITY_REQUIRED is already running 2) Additional check for containers to be on already requested nodes and discarding them during restart Needless to say, this patch requires Hadoop 2.7.0 to build. My testing with the Slider-HBase package using REGION_SERVER with ANTI_AFFINITY_REQUIRED flag does seem to function as expected with all the checks added, getting enforced during flex. Also any containers that got allocated on already requested nodes during restart were being discarded. Will investigate and add some unit tests soon. Meanwhile, it will be useful, if someone can review the patch and add their thought/comments.
          Hide
          kartha Rajesh Kartha added a comment -

          Found one issue with my patch, should check for nulls on the roleHistory and role objects being passed. Will upload an updated patch soon.

          Also, this patch does not consider escalation of requests. Not sure, if that could be handled via a separate JIRA.

          Show
          kartha Rajesh Kartha added a comment - Found one issue with my patch, should check for nulls on the roleHistory and role objects being passed. Will upload an updated patch soon. Also, this patch does not consider escalation of requests. Not sure, if that could be handled via a separate JIRA.
          Hide
          kartha Rajesh Kartha added a comment -

          Added the null checks for roleHistory and role objects.

          The patch does not cater to escalation

          Also, performing flex for a component without ANTI_AFFINITY_REQUIRED after a flex for a component with ANTI_AFFINITY_REQUIRED does not reset the updateBlacklist() so the container does not get allocated.
          I am yet to come across a way in the YARN API to reset the updateBlacklist() .

          Show
          kartha Rajesh Kartha added a comment - Added the null checks for roleHistory and role objects. The patch does not cater to escalation Also, performing flex for a component without ANTI_AFFINITY_REQUIRED after a flex for a component with ANTI_AFFINITY_REQUIRED does not reset the updateBlacklist() so the container does not get allocated. I am yet to come across a way in the YARN API to reset the updateBlacklist() .
          Hide
          kartha Rajesh Kartha added a comment -

          I know folks are busy. Was wondering if anyone got a chance to try my patch for SLIDER-82 and if there are any thoughts/comments.

          Thanks in advance.
          -Rajesh

          Show
          kartha Rajesh Kartha added a comment - I know folks are busy. Was wondering if anyone got a chance to try my patch for SLIDER-82 and if there are any thoughts/comments. Thanks in advance. -Rajesh
          Hide
          stevel@apache.org Steve Loughran added a comment -

          sorry, not had a chance yet. getting slider to work against Hadoop 2.7.1 and some spark+kerberos fun have been keeping me tied down

          has anyone else had a look?

          Show
          stevel@apache.org Steve Loughran added a comment - sorry, not had a chance yet. getting slider to work against Hadoop 2.7.1 and some spark+kerberos fun have been keeping me tied down has anyone else had a look?
          Hide
          gsaha Gour Saha added a comment -

          Rajesh Kartha can you add some tests?

          Show
          gsaha Gour Saha added a comment - Rajesh Kartha can you add some tests?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Needs Hadoop 2.7.1 to work, so tagging as depending on SLIDER-936

          Show
          stevel@apache.org Steve Loughran added a comment - Needs Hadoop 2.7.1 to work, so tagging as depending on SLIDER-936
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 12893b96bc2e95e1b1ef1aba2ecf39b1330b6a32 in incubator-slider's branch refs/heads/feature/SLIDER-82_ANTI_AFFINITY_REQUIRED from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=12893b9 ]

          SLIDER-82 support anti-affinity: this is the original submission with some minor tweaks and reformatting

          Show
          jira-bot ASF subversion and git services added a comment - Commit 12893b96bc2e95e1b1ef1aba2ecf39b1330b6a32 in incubator-slider's branch refs/heads/feature/ SLIDER-82 _ANTI_AFFINITY_REQUIRED from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=12893b9 ] SLIDER-82 support anti-affinity: this is the original submission with some minor tweaks and reformatting
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 8d0e5b71e4e1758a2c96fc41736f452d496b4f2b in incubator-slider's branch refs/heads/feature/SLIDER-82_ANTI_AFFINITY_REQUIRED from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=8d0e5b7 ]

          Minor cleanup/tuning alongside SLIDER-82

          Show
          jira-bot ASF subversion and git services added a comment - Commit 8d0e5b71e4e1758a2c96fc41736f452d496b4f2b in incubator-slider's branch refs/heads/feature/ SLIDER-82 _ANTI_AFFINITY_REQUIRED from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=8d0e5b7 ] Minor cleanup/tuning alongside SLIDER-82
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit f43fb55ad5e15fddff446142c26eb43a8b226397 in incubator-slider's branch refs/heads/feature/SLIDER-82_ANTI_AFFINITY_REQUIRED from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=f43fb55 ]

          SLIDER-82 support anti-affinity: this is the original submission with some minor tweaks and reformatting

          Show
          jira-bot ASF subversion and git services added a comment - Commit f43fb55ad5e15fddff446142c26eb43a8b226397 in incubator-slider's branch refs/heads/feature/ SLIDER-82 _ANTI_AFFINITY_REQUIRED from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=f43fb55 ] SLIDER-82 support anti-affinity: this is the original submission with some minor tweaks and reformatting
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Hi

          I've looked at this, it's now in the git repo as feature/SLIDER-82_ANTI_AFFINITY_REQUIRED

          I can see that it works, but not very well, and may not work if there is >1 role trying to be placed.

          The blacklist isn't per priority, it's for every role: you can't request >1 role type at the same time. Specifically,
          if I was trying to place role "hbase worker" and had blacklisted all but one node, a request for hbase master may pick up
          that same blacklist, and not get placed.

          Even if it was per-role, we can't stop >1 node being allocated on the same container.

          What I do like is the brute force "reject all non-affine allocations" in onContainerAllocated(). It's inefficient, but, provided we are allocated
          containers on different nodes, will succeed.

          Two problems with it

          • there are no guaranteed in the scheduler that you don't get the same one back; that blacklisting
            is essential.
          • premption means that being given those containers that are then releases is very expensive: other
            people's work is lost.

          Which makes me realise that yes, you do need to use the blacklist —as your code does.

          Anyway, I don't see that we can do it as is. Without YARN helping, the only way that
          we can be sure things work is if we ask for exactly one instance of one role at a time.

          The algorithm would be:

          loop through each role

          for each role with requests to make:
          blacklist all nodes which are either failed or hosting an instance already
          request exactly one new node
          wait for that allocation before continuing to ask for another instance or the next role

          It would be slow, indeed, if container requests could not be satisfied for one role, then all other roles could block. But it would ensure that allocated nodes would be anti-affine.

          Show
          stevel@apache.org Steve Loughran added a comment - Hi I've looked at this, it's now in the git repo as feature/ SLIDER-82 _ANTI_AFFINITY_REQUIRED I can see that it works, but not very well, and may not work if there is >1 role trying to be placed. The blacklist isn't per priority, it's for every role: you can't request >1 role type at the same time. Specifically, if I was trying to place role "hbase worker" and had blacklisted all but one node, a request for hbase master may pick up that same blacklist, and not get placed. Even if it was per-role, we can't stop >1 node being allocated on the same container. What I do like is the brute force "reject all non-affine allocations" in onContainerAllocated(). It's inefficient, but, provided we are allocated containers on different nodes, will succeed. Two problems with it there are no guaranteed in the scheduler that you don't get the same one back; that blacklisting is essential. premption means that being given those containers that are then releases is very expensive: other people's work is lost. Which makes me realise that yes, you do need to use the blacklist —as your code does. Anyway, I don't see that we can do it as is. Without YARN helping, the only way that we can be sure things work is if we ask for exactly one instance of one role at a time. The algorithm would be: loop through each role for each role with requests to make: blacklist all nodes which are either failed or hosting an instance already request exactly one new node wait for that allocation before continuing to ask for another instance or the next role It would be slow, indeed, if container requests could not be satisfied for one role, then all other roles could block. But it would ensure that allocated nodes would be anti-affine.
          Hide
          sershe Sergey Shelukhin added a comment -

          Should this be pushed to YARN then? If YARN chooses the locations it seems like YARN should enforce anti-affinity.

          Show
          sershe Sergey Shelukhin added a comment - Should this be pushed to YARN then? If YARN chooses the locations it seems like YARN should enforce anti-affinity.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Yes, that's what YARN-1042 is for. What we are looking at here is what can we do without waiting...

          Show
          stevel@apache.org Steve Loughran added a comment - Yes, that's what YARN-1042 is for. What we are looking at here is what can we do without waiting...
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 8d0e5b71e4e1758a2c96fc41736f452d496b4f2b in incubator-slider's branch refs/heads/develop from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=8d0e5b7 ]

          Minor cleanup/tuning alongside SLIDER-82

          Show
          jira-bot ASF subversion and git services added a comment - Commit 8d0e5b71e4e1758a2c96fc41736f452d496b4f2b in incubator-slider's branch refs/heads/develop from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=8d0e5b7 ] Minor cleanup/tuning alongside SLIDER-82
          Hide
          stevel@apache.org Steve Loughran added a comment -

          This is what I'm thinking here

          For roles with anti-affinity enabled, we request them one-by one, with a node map for the request which lists every node in the cluster which doesn't have an instance.

          As such requests are independent of requests at other priorities, other roles can come up simultaneously —we don't have to play with blacklists

          Algorithm

          Algorithm starts off very similar to the -002 patch, except now we're using node maps rather than blacklists, and so can work in parallel. I'll try to start with that code.

          1. for anti-affinity placements, have a flag on the role status to indicate "anti-affinity request in progress", and a counter of pending. These need to be used when reviewing sizes of components
          2. when reviewing such a component, requests for placed instances are made first, and pushed out in parallel.
          3. next, the current node map is requested
          4. Exactly one container request is made, asking for anywhere on the entire set of nodes for which there is neither a live container, nor a placed request. Relax=false. This is the anti-affinity request
          5. If placed requests are satisfied, launch containers there.
          6. If the outstanding anti-affinity request is satisfied, launch a container on that node, then issue a new anti-affinity request with that node also excluded from the set of possible nodes.
          7. if placed requests cannot be satisfied, the escalation policy will have to become "just another anti-affinity request". Drop the requested node from the list of historical placements, so that it can be included in future anti-affinity requests (but not any outstanding one, which will be left alone).

          With this algorithm, once there's a placement history, requests for instances on those historical nodes will be issued in parallel, and satisfied if the nodes have capacity. It's only instances for which there is no placement history (including escalated ones), where the time to build up the components is O(unplaced-components). As example, bringing up 3 kafka nodes -if we remember where they were, we just ask for them back there again and wait. If there's not any/enough history, then the one-by-one anti-affinity request process happens for all the ones we don't have a place for. For escalation, the same process will apply —but only after the escalation timeout for that component kicks in.

          Fun little details

          1. to handle affinity + placement history, we'll have to run through all nodes with a historical placement first, then escalate them to anti-affinity requests.
          2. the current model of counting nodes by: live + outstanding + delete-in-progress will need to be extended to add a queued-for-affinity field; when flexing down a cluster, these will be dropped even before cancelling outstanding requests.
          3. What if new nodes join a cluster while a request is outstanding? Plan: don't even check for this; successor requests will refresh the node map.
          4. What to do when anti-affinity placement requests cannot be satisfied? Plan: leave them outstanding. This introduces a specific situation: an application cannot become quorate with anti-affinity set, even if there's capacity in the cluster. This is something that could be added to monitoring later -such as allowing the app to fail if it cannot go quorate in a specified time. (of course, that adds a follow-in problem: what if, after failures, the min number of nodes cannot be satisfied.
          5. what if, while there is a pending anti-affinity request, one of the live cluster nodes fails? Really, that should trigger a re-request on same node + escalation plan. Probably best done by cancelling the outstanding request and then restarting the review & request process, so the placed request goes out first.
          6. cluster restart. Outstanding requests get lost, so rebuild state and carry on from there, issuing new placed and anti-affine requests.
          7. what if you want anti-affine but not placement history? Skip the placed phase. This may actually prove slower, as you can't issue requests in parallel. But: probability of request satisfaction is higher; there'll be no timeouts or escalation process.
          8. what if you only want one instance? This is covered implicitly: just issue a single anti-affine request (i.e. we don't special case it). This won't hurt and it ensures a single code path for testing.

          Testing

          we could do some minimal stuff with the mock RM. This doesn't test YARN. Minicluster doesn't work, as there's only one host: localhost.

          That leaves integration tests with clusters of sizes > 1. We can do this, we'll just need to skip those tests if cluster size < 2.

          Possible tests:

          • request 2 anti-affine containers with no history; expect instances on both nodes
          • request 2 anti-affine containers with a history: expect instances on both nodes. We may want to add some way to get the placement history so we can determine whether the requests were satisfied through history vs anti-affine.
          • request 2 anti-affine containers with a history for one. expect instances on both nodes
          • request 1 anti-affine container with a history for a node that lacks capacity/is offline. Expect: escalation to another node.
          • request more anti-affine containers than the cluster has nodes. Expect: unsatisfied requests.
            -AM restart while unsatisfied request is outstanding. Expect: same state is eventually reached.
            -kill container while outstanding request. Expect: request re-issued for that container, then new unsatisfiable request issued
          Show
          stevel@apache.org Steve Loughran added a comment - This is what I'm thinking here For roles with anti-affinity enabled, we request them one-by one, with a node map for the request which lists every node in the cluster which doesn't have an instance. As such requests are independent of requests at other priorities, other roles can come up simultaneously —we don't have to play with blacklists Algorithm Algorithm starts off very similar to the -002 patch, except now we're using node maps rather than blacklists, and so can work in parallel. I'll try to start with that code. for anti-affinity placements, have a flag on the role status to indicate "anti-affinity request in progress", and a counter of pending. These need to be used when reviewing sizes of components when reviewing such a component, requests for placed instances are made first, and pushed out in parallel. next, the current node map is requested Exactly one container request is made, asking for anywhere on the entire set of nodes for which there is neither a live container, nor a placed request. Relax=false. This is the anti-affinity request If placed requests are satisfied, launch containers there. If the outstanding anti-affinity request is satisfied, launch a container on that node, then issue a new anti-affinity request with that node also excluded from the set of possible nodes. if placed requests cannot be satisfied, the escalation policy will have to become "just another anti-affinity request". Drop the requested node from the list of historical placements, so that it can be included in future anti-affinity requests (but not any outstanding one, which will be left alone). With this algorithm, once there's a placement history, requests for instances on those historical nodes will be issued in parallel, and satisfied if the nodes have capacity. It's only instances for which there is no placement history (including escalated ones), where the time to build up the components is O(unplaced-components) . As example, bringing up 3 kafka nodes -if we remember where they were, we just ask for them back there again and wait. If there's not any/enough history, then the one-by-one anti-affinity request process happens for all the ones we don't have a place for. For escalation, the same process will apply —but only after the escalation timeout for that component kicks in. Fun little details to handle affinity + placement history, we'll have to run through all nodes with a historical placement first, then escalate them to anti-affinity requests. the current model of counting nodes by: live + outstanding + delete-in-progress will need to be extended to add a queued-for-affinity field; when flexing down a cluster, these will be dropped even before cancelling outstanding requests. What if new nodes join a cluster while a request is outstanding? Plan: don't even check for this; successor requests will refresh the node map. What to do when anti-affinity placement requests cannot be satisfied? Plan: leave them outstanding. This introduces a specific situation: an application cannot become quorate with anti-affinity set, even if there's capacity in the cluster. This is something that could be added to monitoring later -such as allowing the app to fail if it cannot go quorate in a specified time. (of course, that adds a follow-in problem: what if, after failures, the min number of nodes cannot be satisfied. what if, while there is a pending anti-affinity request, one of the live cluster nodes fails? Really, that should trigger a re-request on same node + escalation plan. Probably best done by cancelling the outstanding request and then restarting the review & request process, so the placed request goes out first. cluster restart. Outstanding requests get lost, so rebuild state and carry on from there, issuing new placed and anti-affine requests. what if you want anti-affine but not placement history? Skip the placed phase. This may actually prove slower, as you can't issue requests in parallel. But: probability of request satisfaction is higher; there'll be no timeouts or escalation process. what if you only want one instance? This is covered implicitly: just issue a single anti-affine request (i.e. we don't special case it). This won't hurt and it ensures a single code path for testing. Testing we could do some minimal stuff with the mock RM. This doesn't test YARN. Minicluster doesn't work, as there's only one host: localhost. That leaves integration tests with clusters of sizes > 1. We can do this, we'll just need to skip those tests if cluster size < 2. Possible tests: request 2 anti-affine containers with no history; expect instances on both nodes request 2 anti-affine containers with a history: expect instances on both nodes. We may want to add some way to get the placement history so we can determine whether the requests were satisfied through history vs anti-affine. request 2 anti-affine containers with a history for one. expect instances on both nodes request 1 anti-affine container with a history for a node that lacks capacity/is offline. Expect: escalation to another node. request more anti-affine containers than the cluster has nodes. Expect: unsatisfied requests. -AM restart while unsatisfied request is outstanding. Expect: same state is eventually reached. -kill container while outstanding request. Expect: request re-issued for that container, then new unsatisfiable request issued
          Hide
          sherryxg Sherry Guo added a comment -

          Hi Steve,
          Can I work on this if no one is currently working on it?

          Show
          sherryxg Sherry Guo added a comment - Hi Steve, Can I work on this if no one is currently working on it?
          Hide
          sherryxg Sherry Guo added a comment -

          I forgot to mention that Rajesh, who was originally working on this has taken on a different role and I am picking up his work...

          Show
          sherryxg Sherry Guo added a comment - I forgot to mention that Rajesh, who was originally working on this has taken on a different role and I am picking up his work...
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I've got it on my list of things to do next week, at least the core algorithm. Where I would like help is

          -reviewing: algorithm, code
          -writing tests
          -running tests

          BTW, one extra change to algorithm, step "1.5": get the node map, and skip those nodes which are down for a placed request. That will avoid the delay for escalation when a node is visibly down.

          Show
          stevel@apache.org Steve Loughran added a comment - I've got it on my list of things to do next week, at least the core algorithm. Where I would like help is -reviewing: algorithm, code -writing tests -running tests BTW, one extra change to algorithm, step "1.5": get the node map, and skip those nodes which are down for a placed request. That will avoid the delay for escalation when a node is visibly down.
          Hide
          sherryxg Sherry Guo added a comment -

          I'm happy to help in any way I can. I am new to Slider and the open source community and I am interested in contributing. If there's anything small for a beginner, please let me know.

          By the way, I noticed that the develop build is broken.

          Thanks,

          Show
          sherryxg Sherry Guo added a comment - I'm happy to help in any way I can. I am new to Slider and the open source community and I am interested in contributing. If there's anything small for a beginner, please let me know. By the way, I noticed that the develop build is broken. Thanks,
          Hide
          gsaha Gour Saha added a comment -

          Develop is fixed now. Just refresh and retry.

          Show
          gsaha Gour Saha added a comment - Develop is fixed now. Just refresh and retry.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Extra complication: labels.

          When a list of possible target nodes is calculated, it must only include those which hold the labels associated with that role.

          example, role "master" needs the label "static". After enumerating all nodes in the cluster, all nodes which don't have that label must be stripped prior to building the request. Otherwise you get an inconsistent constraint.

          Show
          stevel@apache.org Steve Loughran added a comment - Extra complication: labels. When a list of possible target nodes is calculated, it must only include those which hold the labels associated with that role. example, role "master" needs the label "static". After enumerating all nodes in the cluster, all nodes which don't have that label must be stripped prior to building the request. Otherwise you get an inconsistent constraint.
          Hide
          thw Thomas Weise added a comment -

          Since the new approach relies on requesting specific nodes instead of blacklisting: Working on Apache Apex we have seen issues requesting specific nodes on CDH up till the last release. We have seen the same issue testing KOYA on CDH with component recovery:

          https://mail-archives.apache.org/mod_mbox/incubator-slider-dev/201505.mbox/%3CD1823655.E319%25gsaha@hortonworks.com%3E

          I can volunteer to test the patch on CDH.

          Show
          thw Thomas Weise added a comment - Since the new approach relies on requesting specific nodes instead of blacklisting: Working on Apache Apex we have seen issues requesting specific nodes on CDH up till the last release. We have seen the same issue testing KOYA on CDH with component recovery: https://mail-archives.apache.org/mod_mbox/incubator-slider-dev/201505.mbox/%3CD1823655.E319%25gsaha@hortonworks.com%3E I can volunteer to test the patch on CDH.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit bf4a3972951056862dfecfe0ba625aec041d96c4 in incubator-slider's branch refs/heads/feature/SLIDER-82-anti-affinity-attempt-2 from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=bf4a397 ]

          SLIDER-82 preparation: java-7 <>-ify some structures

          Show
          jira-bot ASF subversion and git services added a comment - Commit bf4a3972951056862dfecfe0ba625aec041d96c4 in incubator-slider's branch refs/heads/feature/ SLIDER-82 -anti-affinity-attempt-2 from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=bf4a397 ] SLIDER-82 preparation: java-7 <>-ify some structures
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit de040cbb87eb8a55bf9eb399422a48266c937ee7 in incubator-slider's branch refs/heads/feature/SLIDER-82-anti-affinity-attempt-2 from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=de040cb ]

          SLIDER-82 SLIDER-947 build node map from yarn update reports; serve via REST/IPC

          Show
          jira-bot ASF subversion and git services added a comment - Commit de040cbb87eb8a55bf9eb399422a48266c937ee7 in incubator-slider's branch refs/heads/feature/ SLIDER-82 -anti-affinity-attempt-2 from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=de040cb ] SLIDER-82 SLIDER-947 build node map from yarn update reports; serve via REST/IPC
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 88a7b34cacbbe7b092b7c02dbe1e653772fc441e in incubator-slider's branch refs/heads/feature/SLIDER-82-anti-affinity-attempt-2 from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=88a7b34 ]

          SLIDER-82 IDE suggested AM cleanup

          Show
          jira-bot ASF subversion and git services added a comment - Commit 88a7b34cacbbe7b092b7c02dbe1e653772fc441e in incubator-slider's branch refs/heads/feature/ SLIDER-82 -anti-affinity-attempt-2 from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=88a7b34 ] SLIDER-82 IDE suggested AM cleanup
          Hide
          stevel@apache.org Steve Loughran added a comment -

          It turns out that the AM doesn't get a node map via the node map update callback except when the NM changes, and you don't get an initial map on AM registration (which would be the obvious place).

          What to do?

          1. 2.8+ get the RM/AM protocol to include the list of nodes on registration. Best long term strategy for all. Slider on Hadoop <2.8 would have to handle the missing list, perhaps by not handling anti-affinity there.
          2. get the node list from the YARN client on launch. Works for short-lived services, and could be maintained for the life of the AM —but would be lost on AM restart.
          3. bring up a YARN client in the RM. Not completely implausible, as the keytab to do this will come up, and for keytabless operation, the slider client could ask for a token to talk to the RM as the principal.
          Show
          stevel@apache.org Steve Loughran added a comment - It turns out that the AM doesn't get a node map via the node map update callback except when the NM changes, and you don't get an initial map on AM registration (which would be the obvious place). What to do? 2.8+ get the RM/AM protocol to include the list of nodes on registration. Best long term strategy for all. Slider on Hadoop <2.8 would have to handle the missing list, perhaps by not handling anti-affinity there. get the node list from the YARN client on launch. Works for short-lived services, and could be maintained for the life of the AM —but would be lost on AM restart. bring up a YARN client in the RM. Not completely implausible, as the keytab to do this will come up, and for keytabless operation, the slider client could ask for a token to talk to the RM as the principal.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          that sounds like some scheduler bug. Do you know if there's been a YARN jira on it?

          Show
          stevel@apache.org Steve Loughran added a comment - that sounds like some scheduler bug. Do you know if there's been a YARN jira on it?
          Hide
          thw Thomas Weise added a comment - - edited

          Actually there is:

          https://issues.apache.org/jira/browse/YARN-1412

          There is a separate app to reproduce linked here:

          https://malhar.atlassian.net/browse/APEX-123

          Show
          thw Thomas Weise added a comment - - edited Actually there is: https://issues.apache.org/jira/browse/YARN-1412 There is a separate app to reproduce linked here: https://malhar.atlassian.net/browse/APEX-123
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I should note that YARN-371 implies as there'll be 1 request issued for every node listed, this may be a fairly expensive operation on a large cluster.

          Show
          stevel@apache.org Steve Loughran added a comment - I should note that YARN-371 implies as there'll be 1 request issued for every node listed, this may be a fairly expensive operation on a large cluster.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 65617d51be32f2e5c48143b2c5c71681cc946aca in incubator-slider's branch refs/heads/feature/SLIDER-82-anti-affinity-attempt-2 from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=65617d5 ]

          Merge branch 'develop' into feature/SLIDER-82-anti-affinity-attempt-2

          Show
          jira-bot ASF subversion and git services added a comment - Commit 65617d51be32f2e5c48143b2c5c71681cc946aca in incubator-slider's branch refs/heads/feature/ SLIDER-82 -anti-affinity-attempt-2 from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=65617d5 ] Merge branch 'develop' into feature/ SLIDER-82 -anti-affinity-attempt-2
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 002a88a05a469ac547c79095a81991a47661ac89 in incubator-slider's branch refs/heads/feature/SLIDER-82-anti-affinity-attempt-2 from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=002a88a ]

          SLIDER-82 disable nodes updated check in TestRESTStandalone; allows for feature branch to be resynced into develop/

          Show
          jira-bot ASF subversion and git services added a comment - Commit 002a88a05a469ac547c79095a81991a47661ac89 in incubator-slider's branch refs/heads/feature/ SLIDER-82 -anti-affinity-attempt-2 from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=002a88a ] SLIDER-82 disable nodes updated check in TestRESTStandalone; allows for feature branch to be resynced into develop/
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 88a7b34cacbbe7b092b7c02dbe1e653772fc441e in incubator-slider's branch refs/heads/develop from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=88a7b34 ]

          SLIDER-82 IDE suggested AM cleanup

          Show
          jira-bot ASF subversion and git services added a comment - Commit 88a7b34cacbbe7b092b7c02dbe1e653772fc441e in incubator-slider's branch refs/heads/develop from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=88a7b34 ] SLIDER-82 IDE suggested AM cleanup
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 65617d51be32f2e5c48143b2c5c71681cc946aca in incubator-slider's branch refs/heads/develop from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=65617d5 ]

          Merge branch 'develop' into feature/SLIDER-82-anti-affinity-attempt-2

          Show
          jira-bot ASF subversion and git services added a comment - Commit 65617d51be32f2e5c48143b2c5c71681cc946aca in incubator-slider's branch refs/heads/develop from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=65617d5 ] Merge branch 'develop' into feature/ SLIDER-82 -anti-affinity-attempt-2
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 002a88a05a469ac547c79095a81991a47661ac89 in incubator-slider's branch refs/heads/develop from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=002a88a ]

          SLIDER-82 disable nodes updated check in TestRESTStandalone; allows for feature branch to be resynced into develop/

          Show
          jira-bot ASF subversion and git services added a comment - Commit 002a88a05a469ac547c79095a81991a47661ac89 in incubator-slider's branch refs/heads/develop from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=002a88a ] SLIDER-82 disable nodes updated check in TestRESTStandalone; allows for feature branch to be resynced into develop/
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit c8fb17e051553c3fff737dc65eaa6b3ad0f8563c in incubator-slider's branch refs/heads/develop from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=c8fb17e ]

          Merge branch 'feature/SLIDER-82-anti-affinity-attempt-2' into develop

          Show
          jira-bot ASF subversion and git services added a comment - Commit c8fb17e051553c3fff737dc65eaa6b3ad0f8563c in incubator-slider's branch refs/heads/develop from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=c8fb17e ] Merge branch 'feature/ SLIDER-82 -anti-affinity-attempt-2' into develop
          Hide
          stevel@apache.org Steve Loughran added a comment -

          status: I've just merged the ongoing work into develop. This does not address anti-affinity at all, and SLIDER-947 is outstanding; we'll need to stand up a YarnClient in the AM to bootstrap the state. (mock cluster tests will need to mimic this). I've merged it in to keep delta between feature and app branch low.

          What is in are

          • ongoing patches during the process: SLIDER-948, SLIDER-943
          • All RoleHistory state is now published as metrics. That's part of my plan for metrics-based testing: publish state to metrics, and test runners poll those metrics to determine internal state of app.
          • nodemap is published via REST and IPC.
          Show
          stevel@apache.org Steve Loughran added a comment - status: I've just merged the ongoing work into develop. This does not address anti-affinity at all, and SLIDER-947 is outstanding; we'll need to stand up a YarnClient in the AM to bootstrap the state. (mock cluster tests will need to mimic this). I've merged it in to keep delta between feature and app branch low. What is in are ongoing patches during the process: SLIDER-948 , SLIDER-943 All RoleHistory state is now published as metrics. That's part of my plan for metrics-based testing: publish state to metrics, and test runners poll those metrics to determine internal state of app. nodemap is published via REST and IPC.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 4e9a95b926b0b1bae2d210f63fcb592043be6091 in incubator-slider's branch refs/heads/feature/SLIDER-82-pass-3 from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=4e9a95b ]

          SLIDER-82 preparing ground for anti-affinity.

          • app state binding moves from multiple args to a single binding struct; this significantly simplifies test setup
          • references in roleHistory to available nodes are replace with "recent", as that is what they currently are
          Show
          jira-bot ASF subversion and git services added a comment - Commit 4e9a95b926b0b1bae2d210f63fcb592043be6091 in incubator-slider's branch refs/heads/feature/ SLIDER-82 -pass-3 from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=4e9a95b ] SLIDER-82 preparing ground for anti-affinity. app state binding moves from multiple args to a single binding struct; this significantly simplifies test setup references in roleHistory to available nodes are replace with "recent", as that is what they currently are
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit da24357a4de4cdf1c494eaf830441fe49bf48355 in incubator-slider's branch refs/heads/feature/SLIDER-82-pass-3 from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=da24357 ]

          SLIDER-82 minor test source cleanup

          Show
          jira-bot ASF subversion and git services added a comment - Commit da24357a4de4cdf1c494eaf830441fe49bf48355 in incubator-slider's branch refs/heads/feature/ SLIDER-82 -pass-3 from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=da24357 ] SLIDER-82 minor test source cleanup
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit c6231fe769190d6da21c77e6c74298b08f49319b in incubator-slider's branch refs/heads/feature/SLIDER-82-pass-3 from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=c6231fe ]

          SLIDER-82 setting up node listings into AppState binding

          Show
          jira-bot ASF subversion and git services added a comment - Commit c6231fe769190d6da21c77e6c74298b08f49319b in incubator-slider's branch refs/heads/feature/ SLIDER-82 -pass-3 from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=c6231fe ] SLIDER-82 setting up node listings into AppState binding
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit dd6073352d7c7b8690098fee19ecef5b9d4b321f in incubator-slider's branch refs/heads/feature/SLIDER-82-pass-3 from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=dd60733 ]

          SLIDER-82 trying to set up YarnClient, having config propagation issues in minicluster (RM client addr is 0.0.0.0)

          Show
          jira-bot ASF subversion and git services added a comment - Commit dd6073352d7c7b8690098fee19ecef5b9d4b321f in incubator-slider's branch refs/heads/feature/ SLIDER-82 -pass-3 from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=dd60733 ] SLIDER-82 trying to set up YarnClient, having config propagation issues in minicluster (RM client addr is 0.0.0.0)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 6b13042c66c0f8d0914ee18f9abddd170f6ef512 in incubator-slider's branch refs/heads/feature/SLIDER-82-pass-3.1 from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=6b13042 ]

          Merge branch 'develop' into feature/SLIDER-82-pass-3.1

          Show
          jira-bot ASF subversion and git services added a comment - Commit 6b13042c66c0f8d0914ee18f9abddd170f6ef512 in incubator-slider's branch refs/heads/feature/ SLIDER-82 -pass-3.1 from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=6b13042 ] Merge branch 'develop' into feature/ SLIDER-82 -pass-3.1
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 076ecb1d00dafa7030e0facade283f41fa86e43b in incubator-slider's branch refs/heads/feature/SLIDER-82-pass-3.1 from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=076ecb1 ]

          SLIDER-82 TestBuildStandaloneAM.testUpgrade is failing in its post-run checks, because the new role is being declared as outstanding. Disabling the check entirely

          Show
          jira-bot ASF subversion and git services added a comment - Commit 076ecb1d00dafa7030e0facade283f41fa86e43b in incubator-slider's branch refs/heads/feature/ SLIDER-82 -pass-3.1 from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=076ecb1 ] SLIDER-82 TestBuildStandaloneAM.testUpgrade is failing in its post-run checks, because the new role is being declared as outstanding. Disabling the check entirely
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 2487fba2d191bd70c04d5620e220ea825c2938a2 in incubator-slider's branch refs/heads/feature/SLIDER-82-pass-3.1 from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=2487fba ]

          Merge branch 'develop' into feature/SLIDER-82-pass-3.1

          1. Conflicts:
          2. slider-core/src/main/java/org/apache/slider/server/appmaster/SliderAppMaster.java
          3. slider-core/src/test/groovy/org/apache/slider/client/TestClientBadArgs.groovy
          Show
          jira-bot ASF subversion and git services added a comment - Commit 2487fba2d191bd70c04d5620e220ea825c2938a2 in incubator-slider's branch refs/heads/feature/ SLIDER-82 -pass-3.1 from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=2487fba ] Merge branch 'develop' into feature/ SLIDER-82 -pass-3.1 Conflicts: slider-core/src/main/java/org/apache/slider/server/appmaster/SliderAppMaster.java slider-core/src/test/groovy/org/apache/slider/client/TestClientBadArgs.groovy
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 076ecb1d00dafa7030e0facade283f41fa86e43b in incubator-slider's branch refs/heads/develop from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=076ecb1 ]

          SLIDER-82 TestBuildStandaloneAM.testUpgrade is failing in its post-run checks, because the new role is being declared as outstanding. Disabling the check entirely

          Show
          jira-bot ASF subversion and git services added a comment - Commit 076ecb1d00dafa7030e0facade283f41fa86e43b in incubator-slider's branch refs/heads/develop from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=076ecb1 ] SLIDER-82 TestBuildStandaloneAM.testUpgrade is failing in its post-run checks, because the new role is being declared as outstanding. Disabling the check entirely
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 2487fba2d191bd70c04d5620e220ea825c2938a2 in incubator-slider's branch refs/heads/develop from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=2487fba ]

          Merge branch 'develop' into feature/SLIDER-82-pass-3.1

          1. Conflicts:
          2. slider-core/src/main/java/org/apache/slider/server/appmaster/SliderAppMaster.java
          3. slider-core/src/test/groovy/org/apache/slider/client/TestClientBadArgs.groovy
          Show
          jira-bot ASF subversion and git services added a comment - Commit 2487fba2d191bd70c04d5620e220ea825c2938a2 in incubator-slider's branch refs/heads/develop from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=2487fba ] Merge branch 'develop' into feature/ SLIDER-82 -pass-3.1 Conflicts: slider-core/src/main/java/org/apache/slider/server/appmaster/SliderAppMaster.java slider-core/src/test/groovy/org/apache/slider/client/TestClientBadArgs.groovy
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit cf00b9a5d8b277d8b0e2dfa1b0e45075900cebf0 in incubator-slider's branch refs/heads/develop from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=cf00b9a ]

          Merge branch 'feature/SLIDER-82-pass-3.1' into develop

          Show
          jira-bot ASF subversion and git services added a comment - Commit cf00b9a5d8b277d8b0e2dfa1b0e45075900cebf0 in incubator-slider's branch refs/heads/develop from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=cf00b9a ] Merge branch 'feature/ SLIDER-82 -pass-3.1' into develop
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 72d1c8a933387b38aca1f716c1532104f647e1c6 in incubator-slider's branch refs/heads/develop from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=72d1c8a ]

          SLIDER-82 catch up with changed parent method

          Show
          jira-bot ASF subversion and git services added a comment - Commit 72d1c8a933387b38aca1f716c1532104f647e1c6 in incubator-slider's branch refs/heads/develop from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=72d1c8a ] SLIDER-82 catch up with changed parent method
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit d73e56e8a63599bc7a4f688dfa9d4bd1cc03d5a3 in incubator-slider's branch refs/heads/develop from Steve Loughran
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=d73e56e ]

          SLIDER-82 tweak name of TestAgentAAEcho test cluster to be testagentaaecho to help find its logs more easily

          Show
          jira-bot ASF subversion and git services added a comment - Commit d73e56e8a63599bc7a4f688dfa9d4bd1cc03d5a3 in incubator-slider's branch refs/heads/develop from Steve Loughran [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=d73e56e ] SLIDER-82 tweak name of TestAgentAAEcho test cluster to be testagentaaecho to help find its logs more easily

            People

            • Assignee:
              stevel@apache.org Steve Loughran
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              1 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 186h
                186h
                Remaining:
                Remaining Estimate - 184h
                184h
                Logged:
                Remaining Estimate - 184h Time Not Required
                0.5h

                  Development

                    Agile