Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5839 Flink Security problem collection
  3. FLINK-5916

make env.java.opts.jobmanager and env.java.opts.taskmanager working in YARN mode

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0
    • Component/s: YARN
    • Labels:
      None

      Description

      Now only env.java.opts works in YARN mode, and it applies both to JM and TM. I'd like to make env.java.opts.jobmanager and env.java.opts.taskmanager working in YARN mode in addition, to support fine grained params setting.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/3415

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3415
          Show
          till.rohrmann Till Rohrmann added a comment - Fixed via https://github.com/apache/flink/commit/e9a5c8629408d0b5e7fa89a072f16f6788f961a2
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user tillrohrmann commented on the issue:

          https://github.com/apache/flink/pull/3415

          Changes look good to me @WangTaoTheTonic. Thanks a lot for your contribution. Will merge the PR once Travis gives green light.

          Show
          githubbot ASF GitHub Bot added a comment - Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/3415 Changes look good to me @WangTaoTheTonic. Thanks a lot for your contribution. Will merge the PR once Travis gives green light.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user WangTaoTheTonic commented on the issue:

          https://github.com/apache/flink/pull/3415

          @tillrohrmann Thanks for the review. I've added the logic checking if the config is non-empty and test cases

          Show
          githubbot ASF GitHub Bot added a comment - Github user WangTaoTheTonic commented on the issue: https://github.com/apache/flink/pull/3415 @tillrohrmann Thanks for the review. I've added the logic checking if the config is non-empty and test cases
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user tillrohrmann commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3415#discussion_r104170895

          — Diff: flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/BootstrapTools.java —
          @@ -360,6 +360,7 @@ public static String getTaskManagerShellCommand(
          "-Xmx" + tmParams.taskManagerHeapSizeMB() + "m " +
          "-XX:MaxDirectMemorySize=" + tmParams.taskManagerDirectMemoryLimitMB() + "m");
          String javaOpts = flinkConfig.getString(CoreOptions.FLINK_JVM_OPTIONS);
          + javaOpts += " " + flinkConfig.getString(CoreOptions.FLINK_TM_JVM_OPTIONS);
          — End diff –

          Can we only add it if the config string is non-empty?

          Show
          githubbot ASF GitHub Bot added a comment - Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/3415#discussion_r104170895 — Diff: flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/BootstrapTools.java — @@ -360,6 +360,7 @@ public static String getTaskManagerShellCommand( "-Xmx" + tmParams.taskManagerHeapSizeMB() + "m " + "-XX:MaxDirectMemorySize=" + tmParams.taskManagerDirectMemoryLimitMB() + "m"); String javaOpts = flinkConfig.getString(CoreOptions.FLINK_JVM_OPTIONS); + javaOpts += " " + flinkConfig.getString(CoreOptions.FLINK_TM_JVM_OPTIONS); — End diff – Can we only add it if the config string is non-empty?
          Hide
          till.rohrmann Till Rohrmann added a comment -

          I think the mesos-taskmanager.sh script does but not the mesos-appmaster.sh, if I'm not mistaken. We should verify and if true fix it.

          Show
          till.rohrmann Till Rohrmann added a comment - I think the mesos-taskmanager.sh script does but not the mesos-appmaster.sh , if I'm not mistaken. We should verify and if true fix it.
          Hide
          eronwright Eron Wright added a comment -

          +1
          I believe this is already implements in Mesos mode, because Flink-on-Mesos use shell scripts to start the JM/TM (similar to standalone mode).

          An alternate solution for YARN may be to also use startup scripts rather than constructing the `java ...` command from scratch in `BootstrapTools`. This would encourage convergence among the various modes.

          Show
          eronwright Eron Wright added a comment - +1 I believe this is already implements in Mesos mode, because Flink-on-Mesos use shell scripts to start the JM/TM (similar to standalone mode). An alternate solution for YARN may be to also use startup scripts rather than constructing the `java ...` command from scratch in `BootstrapTools`. This would encourage convergence among the various modes.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user WangTaoTheTonic opened a pull request:

          https://github.com/apache/flink/pull/3415

          FLINK-5916[YARN]make env.java.opts.jobmanager and env.java.opts.taskmanager working i…

          Now only env.java.opts works in YARN mode, and it applies both to JM and TM.

          It's useful to make env.java.opts.jobmanager and env.java.opts.taskmanager working in YARN mode in addition, to support fine grained params setting.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/WangTaoTheTonic/flink FLINK-5916

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3415.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3415


          commit 63a4e82858109644c8c9f940fd891b9aea11090a
          Author: WangTaoTheTonic <wangtao111@huawei.com>
          Date: 2017-02-25T07:35:19Z

          make env.java.opts.jobmanager and env.java.opts.taskmanager working in YARN mode


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user WangTaoTheTonic opened a pull request: https://github.com/apache/flink/pull/3415 FLINK-5916 [YARN] make env.java.opts.jobmanager and env.java.opts.taskmanager working i… Now only env.java.opts works in YARN mode, and it applies both to JM and TM. It's useful to make env.java.opts.jobmanager and env.java.opts.taskmanager working in YARN mode in addition, to support fine grained params setting. You can merge this pull request into a Git repository by running: $ git pull https://github.com/WangTaoTheTonic/flink FLINK-5916 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3415.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3415 commit 63a4e82858109644c8c9f940fd891b9aea11090a Author: WangTaoTheTonic <wangtao111@huawei.com> Date: 2017-02-25T07:35:19Z make env.java.opts.jobmanager and env.java.opts.taskmanager working in YARN mode

            People

            • Assignee:
              WangTao Tao Wang
              Reporter:
              WangTao Tao Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development