Apache Twill
  1. Apache Twill
  2. TWILL-50

Be able to specify and run containers on different hosts

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.1.0-incubating
    • Fix Version/s: None
    • Component/s: yarn
    • Labels:
      None

      Description

      Currently even the hosts is being specified, containers may still be allocated on a small set of hosts (or even in the same host). It would be good to have a way to have force containers allocate on different hosts.

        Activity

        Hide
        Alvin Wang added a comment -

        MultiHostExample works with 2 slave nodes, but fails with 9. I'm going to debug this and update the review.

        Show
        Alvin Wang added a comment - MultiHostExample works with 2 slave nodes, but fails with 9. I'm going to debug this and update the review.
        Hide
        Alvin Wang added a comment -
        Show
        Alvin Wang added a comment - Review created: https://reviews.apache.org/r/18921/
        Hide
        Andreas Neumann added a comment -

        Alvin, can you please open a review in ReviewBoard?

        Show
        Andreas Neumann added a comment - Alvin, can you please open a review in ReviewBoard?
        Hide
        Alvin Wang added a comment -

        Also, I need to test on 10+ hosts. Only tested with 2 hosts so far.

        Show
        Alvin Wang added a comment - Also, I need to test on 10+ hosts. Only tested with 2 hosts so far.
        Hide
        Alvin Wang added a comment -

        Eventually, we may want to do the following:
        1. Decouple EventHandler.launchTimeout() from EventHandler.handleAcquiredContainer().
        2. Enable the user to specify some configuration option that causes ApplicationMasterService to scatter runnables.
        3. Make sure EventHandler.handleAcquiredContainer() makes sense outside of the context of YARN.

        Show
        Alvin Wang added a comment - Eventually, we may want to do the following: 1. Decouple EventHandler.launchTimeout() from EventHandler.handleAcquiredContainer(). 2. Enable the user to specify some configuration option that causes ApplicationMasterService to scatter runnables. 3. Make sure EventHandler.handleAcquiredContainer() makes sense outside of the context of YARN.
        Hide
        Alvin Wang added a comment -

        Attached multihost.patch, which adds EventHandler.handleAcquiredContainer(). This allows for EventHandler to accept or reject containers, for control over which containers the Twill runnable instances are deployed to. Default implementation of EventHandler.handleContainer() accepts all containers, which is same as the behavior of ApplicationMasterService before this patch.

        Also added twill-examples module, which contains HelloWorld example and MultiHostExample. MultiHostExample demonstrates usage of EventHandler.handleAcquiredContainer() to scatter runnable instances to unique hosts.

        Show
        Alvin Wang added a comment - Attached multihost.patch, which adds EventHandler.handleAcquiredContainer(). This allows for EventHandler to accept or reject containers, for control over which containers the Twill runnable instances are deployed to. Default implementation of EventHandler.handleContainer() accepts all containers, which is same as the behavior of ApplicationMasterService before this patch. Also added twill-examples module, which contains HelloWorld example and MultiHostExample. MultiHostExample demonstrates usage of EventHandler.handleAcquiredContainer() to scatter runnable instances to unique hosts.

          People

          • Assignee:
            Alvin Wang
            Reporter:
            Terence Yim
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development