Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.3.0
    • Component/s: Mesos
    • Labels:
      None

      Description

      Fenzo supports placement constraints for tasks, and operators expose agent attributes to frameworks in the form of attributes about the agent offer.

      It would be extremely helpful in our multi-tenant cluster to be able to make use of this facility.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user sgran opened a pull request:

          https://github.com/apache/flink/pull/3744

          FLINK-6336 Initial commit of mesos placement constraints

          This adds a HostAttrValueConstraint evaluator for fenzo, allowing tasks
          to be placed on agents with matching attributes.

          Signed-off-by: Stephen Gran <stephen.gran@piksel.com>

          Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
          If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html).
          In addition to going through the list, please provide a meaningful description of your changes.

          • [ ] General
          • The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text")
          • The pull request addresses only one issue
          • Each commit in the PR has a meaningful commit message (including the JIRA id)
          • [ ] Documentation
          • Documentation has been added for new functionality
          • Old documentation affected by the pull request has been updated
          • JavaDoc for public methods has been added
          • [ ] Tests & Build
          • Functionality added by the pull request is covered by tests
          • `mvn clean verify` has been executed successfully locally or a Travis build has passed

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/pikselpalette/flink add_constraints

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3744.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3744


          commit c0e180195336fdd030243f27c4a954a28a1f6a1e
          Author: Stephen Gran <stephen.gran@piksel.com>
          Date: 2017-04-20T09:22:08Z

          Initial commit of mesos placement constraints

          This adds a HostAttrValueConstraint evaluator for fenzo, allowing tasks
          to be placed on agents with matching attributes.

          Signed-off-by: Stephen Gran <stephen.gran@piksel.com>


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user sgran opened a pull request: https://github.com/apache/flink/pull/3744 FLINK-6336 Initial commit of mesos placement constraints This adds a HostAttrValueConstraint evaluator for fenzo, allowing tasks to be placed on agents with matching attributes. Signed-off-by: Stephen Gran <stephen.gran@piksel.com> Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration. If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide] ( http://flink.apache.org/how-to-contribute.html ). In addition to going through the list, please provide a meaningful description of your changes. [ ] General The pull request references the related JIRA issue (" [FLINK-XXX] Jira title text") The pull request addresses only one issue Each commit in the PR has a meaningful commit message (including the JIRA id) [ ] Documentation Documentation has been added for new functionality Old documentation affected by the pull request has been updated JavaDoc for public methods has been added [ ] Tests & Build Functionality added by the pull request is covered by tests `mvn clean verify` has been executed successfully locally or a Travis build has passed You can merge this pull request into a Git repository by running: $ git pull https://github.com/pikselpalette/flink add_constraints Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3744.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3744 commit c0e180195336fdd030243f27c4a954a28a1f6a1e Author: Stephen Gran <stephen.gran@piksel.com> Date: 2017-04-20T09:22:08Z Initial commit of mesos placement constraints This adds a HostAttrValueConstraint evaluator for fenzo, allowing tasks to be placed on agents with matching attributes. Signed-off-by: Stephen Gran <stephen.gran@piksel.com>
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user EronWright commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3744#discussion_r112487388

          — Diff: docs/setup/mesos.md —
          @@ -222,6 +222,8 @@ When running Flink with Marathon, the whole Flink cluster including the job mana

          `mesos.initial-tasks`: The initial workers to bring up when the master starts (*DEFAULT*: The number of workers specified at cluster startup).

          +`mesos.constraints.hard.hostattribute`: Constraints for task placement on mesos (*DEFAULT*: None).
          — End diff –

          The description seems too generic; I suggest `Constraints for task placement on mesos based on host attributes`. An inline example would also be nice, or at least some description of the value format.

          Show
          githubbot ASF GitHub Bot added a comment - Github user EronWright commented on a diff in the pull request: https://github.com/apache/flink/pull/3744#discussion_r112487388 — Diff: docs/setup/mesos.md — @@ -222,6 +222,8 @@ When running Flink with Marathon, the whole Flink cluster including the job mana `mesos.initial-tasks`: The initial workers to bring up when the master starts (* DEFAULT *: The number of workers specified at cluster startup). +`mesos.constraints.hard.hostattribute`: Constraints for task placement on mesos (* DEFAULT *: None). — End diff – The description seems too generic; I suggest `Constraints for task placement on mesos based on host attributes`. An inline example would also be nice, or at least some description of the value format.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user EronWright commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3744#discussion_r112486557

          — Diff: flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java —
          @@ -487,6 +487,11 @@
          public static final String MESOS_INITIAL_TASKS = "mesos.initial-tasks";

          /**
          + * Constraints for placement of Mesos tasks.
          + */
          + public static final String MESOS_CONSTRAINTS = "mesos.constraints.hard.hostattribute";
          — End diff –

          Please remove this constant, since you're using the new ConfigOption approach which doesn't necessitate an entry here.

          Show
          githubbot ASF GitHub Bot added a comment - Github user EronWright commented on a diff in the pull request: https://github.com/apache/flink/pull/3744#discussion_r112486557 — Diff: flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java — @@ -487,6 +487,11 @@ public static final String MESOS_INITIAL_TASKS = "mesos.initial-tasks"; /** + * Constraints for placement of Mesos tasks. + */ + public static final String MESOS_CONSTRAINTS = "mesos.constraints.hard.hostattribute"; — End diff – Please remove this constant, since you're using the new ConfigOption approach which doesn't necessitate an entry here.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user sgran closed the pull request at:

          https://github.com/apache/flink/pull/3744

          Show
          githubbot ASF GitHub Bot added a comment - Github user sgran closed the pull request at: https://github.com/apache/flink/pull/3744
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user sgran reopened a pull request:

          https://github.com/apache/flink/pull/3744

          FLINK-6336 Initial commit of mesos placement constraints

          This adds a HostAttrValueConstraint evaluator for fenzo, allowing tasks
          to be placed on agents with matching attributes.

          Signed-off-by: Stephen Gran <stephen.gran@piksel.com>

          Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
          If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html).
          In addition to going through the list, please provide a meaningful description of your changes.

          • [x] General
          • The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text")
          • The pull request addresses only one issue
          • Each commit in the PR has a meaningful commit message (including the JIRA id)
          • [x] Documentation
          • Documentation has been added for new functionality
          • Old documentation affected by the pull request has been updated
          • JavaDoc for public methods has been added
          • [x] Tests & Build
          • Functionality added by the pull request is covered by tests
          • `mvn clean verify` has been executed successfully locally or a Travis build has passed

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/pikselpalette/flink add_constraints

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3744.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3744



          Show
          githubbot ASF GitHub Bot added a comment - GitHub user sgran reopened a pull request: https://github.com/apache/flink/pull/3744 FLINK-6336 Initial commit of mesos placement constraints This adds a HostAttrValueConstraint evaluator for fenzo, allowing tasks to be placed on agents with matching attributes. Signed-off-by: Stephen Gran <stephen.gran@piksel.com> Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration. If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide] ( http://flink.apache.org/how-to-contribute.html ). In addition to going through the list, please provide a meaningful description of your changes. [x] General The pull request references the related JIRA issue (" [FLINK-XXX] Jira title text") The pull request addresses only one issue Each commit in the PR has a meaningful commit message (including the JIRA id) [x] Documentation Documentation has been added for new functionality Old documentation affected by the pull request has been updated JavaDoc for public methods has been added [x] Tests & Build Functionality added by the pull request is covered by tests `mvn clean verify` has been executed successfully locally or a Travis build has passed You can merge this pull request into a Git repository by running: $ git pull https://github.com/pikselpalette/flink add_constraints Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3744.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3744
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user sgran commented on the issue:

          https://github.com/apache/flink/pull/3744

          I wish there was a way to re-trigger the test suite without closing and reopening the merge request. It seems a bit of a blunt hammer.

          Show
          githubbot ASF GitHub Bot added a comment - Github user sgran commented on the issue: https://github.com/apache/flink/pull/3744 I wish there was a way to re-trigger the test suite without closing and reopening the merge request. It seems a bit of a blunt hammer.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user sgran closed the pull request at:

          https://github.com/apache/flink/pull/3744

          Show
          githubbot ASF GitHub Bot added a comment - Github user sgran closed the pull request at: https://github.com/apache/flink/pull/3744
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user sgran reopened a pull request:

          https://github.com/apache/flink/pull/3744

          FLINK-6336 Initial commit of mesos placement constraints

          This adds a HostAttrValueConstraint evaluator for fenzo, allowing tasks
          to be placed on agents with matching attributes.

          Signed-off-by: Stephen Gran <stephen.gran@piksel.com>

          Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
          If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html).
          In addition to going through the list, please provide a meaningful description of your changes.

          • [x] General
          • The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text")
          • The pull request addresses only one issue
          • Each commit in the PR has a meaningful commit message (including the JIRA id)
          • [x] Documentation
          • Documentation has been added for new functionality
          • Old documentation affected by the pull request has been updated
          • JavaDoc for public methods has been added
          • [x] Tests & Build
          • Functionality added by the pull request is covered by tests
          • `mvn clean verify` has been executed successfully locally or a Travis build has passed

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/pikselpalette/flink add_constraints

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3744.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3744


          commit c0e180195336fdd030243f27c4a954a28a1f6a1e
          Author: Stephen Gran <stephen.gran@piksel.com>
          Date: 2017-04-20T09:22:08Z

          Initial commit of mesos placement constraints

          This adds a HostAttrValueConstraint evaluator for fenzo, allowing tasks
          to be placed on agents with matching attributes.

          Signed-off-by: Stephen Gran <stephen.gran@piksel.com>

          commit 709cfe5bd3464327b8d24322830d20a20be6e557
          Author: Stephen Gran <stephen.gran@piksel.com>
          Date: 2017-04-21T06:17:44Z

          Better docs - add examples

          Signed-off-by: Stephen Gran <stephen.gran@piksel.com>

          commit fc87105ab30c129f70d6cc34724720a3919cde96
          Author: Stephen Gran <stephen.gran@piksel.com>
          Date: 2017-04-21T06:17:47Z

          Drop unnecessary constant

          Signed-off-by: Stephen Gran <stephen.gran@piksel.com>

          commit 154c11930b0c7fd1be5346e67a7486fbf4c3c4ad
          Author: Stephen Gran <stephen.gran@piksel.com>
          Date: 2017-04-21T06:19:04Z

          clarify basis for task placement constraints

          Signed-off-by: Stephen Gran <stephen.gran@piksel.com>

          commit 797736c608cc70448f73f90853440236a093f2f0
          Author: Stephen Gran <stephen.gran@piksel.com>
          Date: 2017-04-21T06:37:50Z

          drop unused import

          Signed-off-by: Stephen Gran <stephen.gran@piksel.com>


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user sgran reopened a pull request: https://github.com/apache/flink/pull/3744 FLINK-6336 Initial commit of mesos placement constraints This adds a HostAttrValueConstraint evaluator for fenzo, allowing tasks to be placed on agents with matching attributes. Signed-off-by: Stephen Gran <stephen.gran@piksel.com> Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration. If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide] ( http://flink.apache.org/how-to-contribute.html ). In addition to going through the list, please provide a meaningful description of your changes. [x] General The pull request references the related JIRA issue (" [FLINK-XXX] Jira title text") The pull request addresses only one issue Each commit in the PR has a meaningful commit message (including the JIRA id) [x] Documentation Documentation has been added for new functionality Old documentation affected by the pull request has been updated JavaDoc for public methods has been added [x] Tests & Build Functionality added by the pull request is covered by tests `mvn clean verify` has been executed successfully locally or a Travis build has passed You can merge this pull request into a Git repository by running: $ git pull https://github.com/pikselpalette/flink add_constraints Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3744.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3744 commit c0e180195336fdd030243f27c4a954a28a1f6a1e Author: Stephen Gran <stephen.gran@piksel.com> Date: 2017-04-20T09:22:08Z Initial commit of mesos placement constraints This adds a HostAttrValueConstraint evaluator for fenzo, allowing tasks to be placed on agents with matching attributes. Signed-off-by: Stephen Gran <stephen.gran@piksel.com> commit 709cfe5bd3464327b8d24322830d20a20be6e557 Author: Stephen Gran <stephen.gran@piksel.com> Date: 2017-04-21T06:17:44Z Better docs - add examples Signed-off-by: Stephen Gran <stephen.gran@piksel.com> commit fc87105ab30c129f70d6cc34724720a3919cde96 Author: Stephen Gran <stephen.gran@piksel.com> Date: 2017-04-21T06:17:47Z Drop unnecessary constant Signed-off-by: Stephen Gran <stephen.gran@piksel.com> commit 154c11930b0c7fd1be5346e67a7486fbf4c3c4ad Author: Stephen Gran <stephen.gran@piksel.com> Date: 2017-04-21T06:19:04Z clarify basis for task placement constraints Signed-off-by: Stephen Gran <stephen.gran@piksel.com> commit 797736c608cc70448f73f90853440236a093f2f0 Author: Stephen Gran <stephen.gran@piksel.com> Date: 2017-04-21T06:37:50Z drop unused import Signed-off-by: Stephen Gran <stephen.gran@piksel.com>
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user sgran closed the pull request at:

          https://github.com/apache/flink/pull/3744

          Show
          githubbot ASF GitHub Bot added a comment - Github user sgran closed the pull request at: https://github.com/apache/flink/pull/3744
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user sgran reopened a pull request:

          https://github.com/apache/flink/pull/3744

          FLINK-6336 Initial commit of mesos placement constraints

          This adds a HostAttrValueConstraint evaluator for fenzo, allowing tasks
          to be placed on agents with matching attributes.

          Signed-off-by: Stephen Gran <stephen.gran@piksel.com>

          Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
          If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html).
          In addition to going through the list, please provide a meaningful description of your changes.

          • [x] General
          • The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text")
          • The pull request addresses only one issue
          • Each commit in the PR has a meaningful commit message (including the JIRA id)
          • [x] Documentation
          • Documentation has been added for new functionality
          • Old documentation affected by the pull request has been updated
          • JavaDoc for public methods has been added
          • [x] Tests & Build
          • Functionality added by the pull request is covered by tests
          • `mvn clean verify` has been executed successfully locally or a Travis build has passed

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/pikselpalette/flink add_constraints

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3744.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3744


          commit c0e180195336fdd030243f27c4a954a28a1f6a1e
          Author: Stephen Gran <stephen.gran@piksel.com>
          Date: 2017-04-20T09:22:08Z

          Initial commit of mesos placement constraints

          This adds a HostAttrValueConstraint evaluator for fenzo, allowing tasks
          to be placed on agents with matching attributes.

          Signed-off-by: Stephen Gran <stephen.gran@piksel.com>

          commit 709cfe5bd3464327b8d24322830d20a20be6e557
          Author: Stephen Gran <stephen.gran@piksel.com>
          Date: 2017-04-21T06:17:44Z

          Better docs - add examples

          Signed-off-by: Stephen Gran <stephen.gran@piksel.com>

          commit fc87105ab30c129f70d6cc34724720a3919cde96
          Author: Stephen Gran <stephen.gran@piksel.com>
          Date: 2017-04-21T06:17:47Z

          Drop unnecessary constant

          Signed-off-by: Stephen Gran <stephen.gran@piksel.com>

          commit 154c11930b0c7fd1be5346e67a7486fbf4c3c4ad
          Author: Stephen Gran <stephen.gran@piksel.com>
          Date: 2017-04-21T06:19:04Z

          clarify basis for task placement constraints

          Signed-off-by: Stephen Gran <stephen.gran@piksel.com>

          commit 797736c608cc70448f73f90853440236a093f2f0
          Author: Stephen Gran <stephen.gran@piksel.com>
          Date: 2017-04-21T06:37:50Z

          drop unused import

          Signed-off-by: Stephen Gran <stephen.gran@piksel.com>


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user sgran reopened a pull request: https://github.com/apache/flink/pull/3744 FLINK-6336 Initial commit of mesos placement constraints This adds a HostAttrValueConstraint evaluator for fenzo, allowing tasks to be placed on agents with matching attributes. Signed-off-by: Stephen Gran <stephen.gran@piksel.com> Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration. If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide] ( http://flink.apache.org/how-to-contribute.html ). In addition to going through the list, please provide a meaningful description of your changes. [x] General The pull request references the related JIRA issue (" [FLINK-XXX] Jira title text") The pull request addresses only one issue Each commit in the PR has a meaningful commit message (including the JIRA id) [x] Documentation Documentation has been added for new functionality Old documentation affected by the pull request has been updated JavaDoc for public methods has been added [x] Tests & Build Functionality added by the pull request is covered by tests `mvn clean verify` has been executed successfully locally or a Travis build has passed You can merge this pull request into a Git repository by running: $ git pull https://github.com/pikselpalette/flink add_constraints Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3744.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3744 commit c0e180195336fdd030243f27c4a954a28a1f6a1e Author: Stephen Gran <stephen.gran@piksel.com> Date: 2017-04-20T09:22:08Z Initial commit of mesos placement constraints This adds a HostAttrValueConstraint evaluator for fenzo, allowing tasks to be placed on agents with matching attributes. Signed-off-by: Stephen Gran <stephen.gran@piksel.com> commit 709cfe5bd3464327b8d24322830d20a20be6e557 Author: Stephen Gran <stephen.gran@piksel.com> Date: 2017-04-21T06:17:44Z Better docs - add examples Signed-off-by: Stephen Gran <stephen.gran@piksel.com> commit fc87105ab30c129f70d6cc34724720a3919cde96 Author: Stephen Gran <stephen.gran@piksel.com> Date: 2017-04-21T06:17:47Z Drop unnecessary constant Signed-off-by: Stephen Gran <stephen.gran@piksel.com> commit 154c11930b0c7fd1be5346e67a7486fbf4c3c4ad Author: Stephen Gran <stephen.gran@piksel.com> Date: 2017-04-21T06:19:04Z clarify basis for task placement constraints Signed-off-by: Stephen Gran <stephen.gran@piksel.com> commit 797736c608cc70448f73f90853440236a093f2f0 Author: Stephen Gran <stephen.gran@piksel.com> Date: 2017-04-21T06:37:50Z drop unused import Signed-off-by: Stephen Gran <stephen.gran@piksel.com>
          Hide
          sgran Stephen Gran added a comment -

          Hi,

          Any chance this can make the 1.3.0 release?

          Thanks!

          Show
          sgran Stephen Gran added a comment - Hi, Any chance this can make the 1.3.0 release? Thanks!
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user tillrohrmann commented on the issue:

          https://github.com/apache/flink/pull/3744

          Thanks for your contribution @sgran and the review @EronWright. Changes look good to me. Merging this PR.

          Show
          githubbot ASF GitHub Bot added a comment - Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/3744 Thanks for your contribution @sgran and the review @EronWright. Changes look good to me. Merging this PR.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/3744

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3744
          Hide
          till.rohrmann Till Rohrmann added a comment -

          Added via 8ef6926f3c735fb4e111379fb1edab5184bc7458

          Show
          till.rohrmann Till Rohrmann added a comment - Added via 8ef6926f3c735fb4e111379fb1edab5184bc7458

            People

            • Assignee:
              sgran Stephen Gran
              Reporter:
              sgran Stephen Gran
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development