Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5153

Allow setting custom application tags for Flink on YARN

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0
    • Component/s: YARN
    • Labels:
      None

      Description

      https://issues.apache.org/jira/browse/YARN-1399 added support in YARN to tag applications.

      We should introduce a configuration variable in Flink allowing users to specify a comma-separated list of tags they want to assign to their Flink on YARN applications.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user patricklucas closed the pull request at:

          https://github.com/apache/flink/pull/3268

          Show
          githubbot ASF GitHub Bot added a comment - Github user patricklucas closed the pull request at: https://github.com/apache/flink/pull/3268
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user rmetzger commented on the issue:

          https://github.com/apache/flink/pull/3268

          @patricklucas Can you manually close the PR? We usually do this automatically, but I forgot to add the right string the commit message.

          Show
          githubbot ASF GitHub Bot added a comment - Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3268 @patricklucas Can you manually close the PR? We usually do this automatically, but I forgot to add the right string the commit message.
          Hide
          rmetzger Robert Metzger added a comment -

          Thanks a lot for your contribution Patrick!

          Change has been merged to 1.3: http://git-wip-us.apache.org/repos/asf/flink/commit/d9c116e5

          Show
          rmetzger Robert Metzger added a comment - Thanks a lot for your contribution Patrick! Change has been merged to 1.3: http://git-wip-us.apache.org/repos/asf/flink/commit/d9c116e5
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user rmetzger commented on the issue:

          https://github.com/apache/flink/pull/3268

          I'll merge the change now.

          I've also tested it.

          Show
          githubbot ASF GitHub Bot added a comment - Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3268 I'll merge the change now. I've also tested it.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user patricklucas commented on the issue:

          https://github.com/apache/flink/pull/3268

          I added the test (which of course also required reflection) to an existing test case. I figured that was the lesser of the two evils of requiring another test application be deployed and hiding a test within a (somewhat) unrelated one.

          It was a bit tricky to actually run the tests, but I verified the correct behavior with both hadoop < 2.4 and >= 2.4.

          Show
          githubbot ASF GitHub Bot added a comment - Github user patricklucas commented on the issue: https://github.com/apache/flink/pull/3268 I added the test (which of course also required reflection) to an existing test case. I figured that was the lesser of the two evils of requiring another test application be deployed and hiding a test within a (somewhat) unrelated one. It was a bit tricky to actually run the tests, but I verified the correct behavior with both hadoop < 2.4 and >= 2.4.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user rmetzger commented on the issue:

          https://github.com/apache/flink/pull/3268

          Hi Patrick,
          the change looks good in general (you updated the documentation as well ).
          One thing I'm missing a bit from the change is a test case.
          We have an entire package (which is, admittedly pretty ugly) for YARN tests.
          Maybe you can just pass your new config option to one of the existing tests and then check if the tag has been set.

          Show
          githubbot ASF GitHub Bot added a comment - Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3268 Hi Patrick, the change looks good in general (you updated the documentation as well ). One thing I'm missing a bit from the change is a test case. We have an entire package (which is, admittedly pretty ugly) for YARN tests. Maybe you can just pass your new config option to one of the existing tests and then check if the tag has been set.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user patricklucas commented on the issue:

          https://github.com/apache/flink/pull/3268

          I manually tested the behavior of this new feature as well as the other YARN application options touched by this change with various Hadoop versions (2.3/2.4/2.6) on a YARN cluster.

          The Travis builds have tended to fail by timing out after 50 min, but everything tests fine locally.

          Show
          githubbot ASF GitHub Bot added a comment - Github user patricklucas commented on the issue: https://github.com/apache/flink/pull/3268 I manually tested the behavior of this new feature as well as the other YARN application options touched by this change with various Hadoop versions (2.3/2.4/2.6) on a YARN cluster. The Travis builds have tended to fail by timing out after 50 min, but everything tests fine locally.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user patricklucas opened a pull request:

          https://github.com/apache/flink/pull/3268

          FLINK-5153 Support YARN application tags

          Adds a new config option `yarn.tags`, a comma-separated list of strings
          passed to YARN as application tags.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/patricklucas/flink FLINK-5153_support_yarn_application_tags

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3268.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3268


          commit b4599e3aa3c10f86354336c258870703e91e1415
          Author: Patrick Lucas <me@patricklucas.com>
          Date: 2017-02-04T00:17:55Z

          FLINK-5153 Support YARN application tags

          Adds a new config option `yarn.tags`, a comma-separated list of strings
          passed to YARN as application tags.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user patricklucas opened a pull request: https://github.com/apache/flink/pull/3268 FLINK-5153 Support YARN application tags Adds a new config option `yarn.tags`, a comma-separated list of strings passed to YARN as application tags. You can merge this pull request into a Git repository by running: $ git pull https://github.com/patricklucas/flink FLINK-5153 _support_yarn_application_tags Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3268.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3268 commit b4599e3aa3c10f86354336c258870703e91e1415 Author: Patrick Lucas <me@patricklucas.com> Date: 2017-02-04T00:17:55Z FLINK-5153 Support YARN application tags Adds a new config option `yarn.tags`, a comma-separated list of strings passed to YARN as application tags.

            People

            • Assignee:
              plucas Patrick Lucas
              Reporter:
              rmetzger Robert Metzger
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development