Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13773

wrong HADOOP_CLIENT_OPTS in hadoop-env on branch-2

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.1, 2.7.3
    • Fix Version/s: 2.8.0
    • Component/s: conf
    • Labels:
      None
    • Target Version/s:

      Description

      in conf/hadoop-env.sh,
      export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS"

      when i set HADOOP_HEAPSIZE ,and run 'hadoop jar ...', jvm args is not work.

      i see, in bin/hadoop,
      exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$@"

      HADOOP_OPTS is behind JAVA_HEAP_MAX, so HADOOP_HEAPSIZE is not work.

      for example i run 'HADOOP_HEAPSIZE=1024 hadoop jar ...' , the java process is 'java -Xmx1024m ... -Xmx512m...', then Xmx512m is valid, and Xmx1024m is invalid

        Issue Links

          Activity

          Hide
          ferhui Fei Hui added a comment -

          in conf/hadoop-env.sh,
          export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS"

          when i set HADOOP_HEAPSIZE ,and run 'hadoop jar ...', jvm args is not work.

          i see, in bin/hadoop,
          exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$@"

          HADOOP_OPTS is behind JAVA_HEAP_MAX, so HADOOP_HEAPSIZE is not work.

          for example i run 'HADOOP_HEAPSIZE=1024 hadoop jar ...' , the java process is 'java -Xmx1024m ... -Xmx512m...', then Xmx512m is valid, and Xmx1024m is invalid

          Show
          ferhui Fei Hui added a comment - in conf/hadoop-env.sh, export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS" when i set HADOOP_HEAPSIZE ,and run 'hadoop jar ...', jvm args is not work. i see, in bin/hadoop, exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$@" HADOOP_OPTS is behind JAVA_HEAP_MAX, so HADOOP_HEAPSIZE is not work. for example i run 'HADOOP_HEAPSIZE=1024 hadoop jar ...' , the java process is 'java -Xmx1024m ... -Xmx512m...', then Xmx512m is valid, and Xmx1024m is invalid
          Hide
          ferhui Fei Hui added a comment -

          it affects branch-2, i will submit a pull request on github

          Show
          ferhui Fei Hui added a comment - it affects branch-2, i will submit a pull request on github
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user ferhui opened a pull request:

          https://github.com/apache/hadoop/pull/150

          HADOOP-13773, set heap args for HADOOP_CLIENT_OPTS when HADOOP_HEAPSI…

          jira url is https://issues.apache.org/jira/browse/HADOOP-13773

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/ferhui/hadoop branch-2

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/hadoop/pull/150.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #150


          commit 1117baf860bf1ddbe3fc18a58bf576e986e3be21
          Author: ferhui <feihui.ustc@gmail.com>
          Date: 2016-10-31T14:52:20Z

          HADOOP-13773, set heap args for HADOOP_CLIENT_OPTS when HADOOP_HEAPSIZE is empty


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user ferhui opened a pull request: https://github.com/apache/hadoop/pull/150 HADOOP-13773 , set heap args for HADOOP_CLIENT_OPTS when HADOOP_HEAPSI… jira url is https://issues.apache.org/jira/browse/HADOOP-13773 You can merge this pull request into a Git repository by running: $ git pull https://github.com/ferhui/hadoop branch-2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/150.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #150 commit 1117baf860bf1ddbe3fc18a58bf576e986e3be21 Author: ferhui <feihui.ustc@gmail.com> Date: 2016-10-31T14:52:20Z HADOOP-13773 , set heap args for HADOOP_CLIENT_OPTS when HADOOP_HEAPSIZE is empty
          Hide
          ferhui Fei Hui added a comment -

          anyone review it ?

          Show
          ferhui Fei Hui added a comment - anyone review it ?
          Hide
          raviprak Ravi Prakash added a comment -

          Hi Fei Hui!

          Welcome to the community and thanks for your contribution! I've taken the liberty to edit the JIRA with values for fields that we are used to. (Fix Version is set when the patch is merged. Target Version is set to the next expected release that would contain the fix. Description contains the problem)

          https://wiki.apache.org/hadoop/HowToContribute is a fairly verbose guide on how to contribute. Instead of making you read it in its entirety, I'd suggest uploading a patch file instead of github pull requests (because what happens to review discussions if Github.com was to fold tomorrow?)

          Just FYI, Allen rewrote the shell scripts for trunk (https://issues.apache.org/jira/browse/HADOOP-9902) and he's the reigning expert in that area. I'll defer to his better judgement. Unfortunately those improvements were not backported fully into branch-2 (to explain any discrepancy you may be seeing)

          It seems in trunk the discrepancy of multiple Xmx values is hired much more elegantly with hadoop_add_param . However to me your fix makes sense for branch-2. We shouldn't be appending -Xmx=512m indiscriminately. Could you please upload the patch file and I'll be happy to commit.

          Show
          raviprak Ravi Prakash added a comment - Hi Fei Hui! Welcome to the community and thanks for your contribution! I've taken the liberty to edit the JIRA with values for fields that we are used to. (Fix Version is set when the patch is merged. Target Version is set to the next expected release that would contain the fix. Description contains the problem) https://wiki.apache.org/hadoop/HowToContribute is a fairly verbose guide on how to contribute. Instead of making you read it in its entirety, I'd suggest uploading a patch file instead of github pull requests (because what happens to review discussions if Github.com was to fold tomorrow?) Just FYI, Allen rewrote the shell scripts for trunk ( https://issues.apache.org/jira/browse/HADOOP-9902 ) and he's the reigning expert in that area. I'll defer to his better judgement. Unfortunately those improvements were not backported fully into branch-2 (to explain any discrepancy you may be seeing) It seems in trunk the discrepancy of multiple Xmx values is hired much more elegantly with hadoop_add_param . However to me your fix makes sense for branch-2. We shouldn't be appending -Xmx=512m indiscriminately. Could you please upload the patch file and I'll be happy to commit.
          Hide
          yuanbo Yuanbo Liu added a comment -

          Fei Hui Thanks for filing this jira.

          suggest uploading a patch file instead of github pull requests....

          Agree with Ravi Prakash, please upload your patch.

          Small suggestion about your code change:

          if [ "$HADOOP_HEAPSIZE" == "" ];
          

          please use "=" instead of "==" here. "==" is not defined in standard POSIX.

          Show
          yuanbo Yuanbo Liu added a comment - Fei Hui Thanks for filing this jira. suggest uploading a patch file instead of github pull requests.... Agree with Ravi Prakash , please upload your patch. Small suggestion about your code change: if [ "$HADOOP_HEAPSIZE" == "" ]; please use "=" instead of "==" here. "==" is not defined in standard POSIX.
          Hide
          ferhui Fei Hui added a comment -

          thanks for Ravi Prakash and Yuanbo Liu 's suggestions, the patch is uploaded
          please review it again.thanks

          Show
          ferhui Fei Hui added a comment - thanks for Ravi Prakash and Yuanbo Liu 's suggestions, the patch is uploaded please review it again.thanks
          Hide
          raviprak Ravi Prakash added a comment -

          Thanks for your contribution Fei Hui and for your careful review Yuanbo Liu!

          I've committed this to branch-2 and branch-2.8.

          Show
          raviprak Ravi Prakash added a comment - Thanks for your contribution Fei Hui and for your careful review Yuanbo Liu! I've committed this to branch-2 and branch-2.8.
          Hide
          raviprak Ravi Prakash added a comment -

          Just for future reference. I've added you to the Contributors1 role. That would allow you to assign issues to yourself. Also, please follow the patch naming scheme. All patch files should have a version. Also, if the patch is not for trunk, it should contain the branch name. So the file name for your patch should be HADOOP-13773.branch-2.01.patch.

          Thanks for your contribution and we look forward to many more from you!

          Show
          raviprak Ravi Prakash added a comment - Just for future reference. I've added you to the Contributors1 role. That would allow you to assign issues to yourself. Also, please follow the patch naming scheme. All patch files should have a version. Also, if the patch is not for trunk, it should contain the branch name. So the file name for your patch should be HADOOP-13773 .branch-2.01.patch. Thanks for your contribution and we look forward to many more from you!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 10s HADOOP-13773 does not apply to branch-2. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HADOOP-13773
          GITHUB PR https://github.com/apache/hadoop/pull/150
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10947/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 10s HADOOP-13773 does not apply to branch-2. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HADOOP-13773 GITHUB PR https://github.com/apache/hadoop/pull/150 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10947/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Resolving since it looks like this was committed to branch-2.8.

          Also it looks like this was committed without a precommit run. Let's try to be more careful in the future.

          Show
          andrew.wang Andrew Wang added a comment - Resolving since it looks like this was committed to branch-2.8. Also it looks like this was committed without a precommit run. Let's try to be more careful in the future.
          Hide
          raviprak Ravi Prakash added a comment -

          Andrew! The precommit was known to fail because the patch applied only to branch-2 (not trunk)

          Show
          raviprak Ravi Prakash added a comment - Andrew! The precommit was known to fail because the patch applied only to branch-2 (not trunk)
          Hide
          andrew.wang Andrew Wang added a comment -

          Hi Ravi, precommit is multi-branch now, if you follow the naming convention of e.g. HADOOP-13733.branch-2.001.patch. See:

          https://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch

          The precommit bot will also detect the branch correctly for github PRs, though if there's both a PR and a patch (as on this JIRA), the precommit bot will always use the PR. When this has happened in the past, I've resorted to filing a new JIRA to unstick precommit.

          Show
          andrew.wang Andrew Wang added a comment - Hi Ravi, precommit is multi-branch now, if you follow the naming convention of e.g. HADOOP-13733 .branch-2.001.patch. See: https://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch The precommit bot will also detect the branch correctly for github PRs, though if there's both a PR and a patch (as on this JIRA), the precommit bot will always use the PR. When this has happened in the past, I've resorted to filing a new JIRA to unstick precommit.
          Hide
          raviprak Ravi Prakash added a comment -

          Awesome! Sounds good Andrew! Thanks. I'll take care in the future.

          Show
          raviprak Ravi Prakash added a comment - Awesome! Sounds good Andrew! Thanks. I'll take care in the future.
          Hide
          ferhui Fei Hui added a comment -

          Thanks. I'll take care in the future.

          Show
          ferhui Fei Hui added a comment - Thanks. I'll take care in the future.

            People

            • Assignee:
              ferhui Fei Hui
              Reporter:
              ferhui Fei Hui
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development