Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11966

Variable cygwin is undefined in hadoop-config.sh when executed through hadoop-daemon.sh.

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 2.7.1, 3.0.0-alpha1
    • Component/s: scripts
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      HADOOP-11464 reinstated support for running the bash scripts through Cygwin. The logic involves setting a cygwin flag variable to indicate if the script is executing through Cygwin. The flag is set in all of the interactive scripts: hadoop, hdfs, yarn and mapred. The flag is not set through hadoop-daemon.sh though. This can cause an erroneous overwrite of HADOOP_HOME and JAVA_LIBRARY_PATH inside hadoop-config.sh.

        Issue Links

          Activity

          Hide
          cnauroth Chris Nauroth added a comment -

          The attached patch refactors setting the cygwin variable into hadoop-config.sh, where it can take effect for all the interactive scripts as well as hadoop-daemon.sh. This is more like how the branch-1 code handled it. Unfortunately, I missed it the first time.

          I have retested Cygwin/Windows with this change. I also retested hadoop-daemon.sh on Linux.

          Show
          cnauroth Chris Nauroth added a comment - The attached patch refactors setting the cygwin variable into hadoop-config.sh, where it can take effect for all the interactive scripts as well as hadoop-daemon.sh. This is more like how the branch-1 code handled it. Unfortunately, I missed it the first time. I have retested Cygwin/Windows with this change. I also retested hadoop-daemon.sh on Linux.
          Hide
          wheat9 Haohui Mai added a comment -

          +1 pending jenkins

          Show
          wheat9 Haohui Mai added a comment - +1 pending jenkins
          Hide
          sanjay.radia Sanjay Radia added a comment -

          You comment says that you tested hadoop-daemon.sh. Have you tested the hadoop, yarn and mapred commands (your change affects all of them and yes it should work since they all source hadoop-config.sh

          Show
          sanjay.radia Sanjay Radia added a comment - You comment says that you tested hadoop-daemon.sh. Have you tested the hadoop, yarn and mapred commands (your change affects all of them and yes it should work since they all source hadoop-config.sh
          Hide
          cnauroth Chris Nauroth added a comment -

          Sanjay Radia, yes, I tested both hadoop-daemon.sh and all 4 of the interactive commands. Thanks!

          Show
          cnauroth Chris Nauroth added a comment - Sanjay Radia , yes, I tested both hadoop-daemon.sh and all 4 of the interactive commands. Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 56s Pre-patch branch-2 compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 5m 49s There were no new javac warning messages.
          +1 javadoc 9m 35s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 shellcheck 0m 5s There were no new shellcheck (v0.3.3) issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 18s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 common tests 23m 8s Tests passed in hadoop-common.
          -1 hdfs tests 165m 18s Tests failed in hadoop-hdfs.
              221m 10s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.tracing.TestTraceAdmin
            hadoop.tools.TestHdfsConfigFields



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732413/HADOOP-11966-branch-2.001.patch
          Optional Tests javadoc javac unit shellcheck
          git revision branch-2 / d5755ba
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6677/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6677/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6677/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6677/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 56s Pre-patch branch-2 compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 5m 49s There were no new javac warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 shellcheck 0m 5s There were no new shellcheck (v0.3.3) issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 18s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 common tests 23m 8s Tests passed in hadoop-common. -1 hdfs tests 165m 18s Tests failed in hadoop-hdfs.     221m 10s   Reason Tests Failed unit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.tracing.TestTraceAdmin   hadoop.tools.TestHdfsConfigFields Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732413/HADOOP-11966-branch-2.001.patch Optional Tests javadoc javac unit shellcheck git revision branch-2 / d5755ba hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6677/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6677/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6677/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6677/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          The HDFS unit test failures are unrelated, because this patch changes shell script code only.

          I committed this to branch-2 and branch-2.7. Haohui and Sanjay, thank you for reviewing.

          Show
          cnauroth Chris Nauroth added a comment - The HDFS unit test failures are unrelated, because this patch changes shell script code only. I committed this to branch-2 and branch-2.7. Haohui and Sanjay, thank you for reviewing.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7817 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7817/)
          HADOOP-11966. Variable cygwin is undefined in hadoop-config.sh when executed through hadoop-daemon.sh. Contributed by Chris Nauroth. (cnauroth: rev 93b770f7e778835a9dd76854b435c5250835d1a8)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7817 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7817/ ) HADOOP-11966 . Variable cygwin is undefined in hadoop-config.sh when executed through hadoop-daemon.sh. Contributed by Chris Nauroth. (cnauroth: rev 93b770f7e778835a9dd76854b435c5250835d1a8) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #927 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/927/)
          HADOOP-11966. Variable cygwin is undefined in hadoop-config.sh when executed through hadoop-daemon.sh. Contributed by Chris Nauroth. (cnauroth: rev 93b770f7e778835a9dd76854b435c5250835d1a8)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #927 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/927/ ) HADOOP-11966 . Variable cygwin is undefined in hadoop-config.sh when executed through hadoop-daemon.sh. Contributed by Chris Nauroth. (cnauroth: rev 93b770f7e778835a9dd76854b435c5250835d1a8) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #196 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/196/)
          HADOOP-11966. Variable cygwin is undefined in hadoop-config.sh when executed through hadoop-daemon.sh. Contributed by Chris Nauroth. (cnauroth: rev 93b770f7e778835a9dd76854b435c5250835d1a8)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #196 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/196/ ) HADOOP-11966 . Variable cygwin is undefined in hadoop-config.sh when executed through hadoop-daemon.sh. Contributed by Chris Nauroth. (cnauroth: rev 93b770f7e778835a9dd76854b435c5250835d1a8) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #195 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/195/)
          HADOOP-11966. Variable cygwin is undefined in hadoop-config.sh when executed through hadoop-daemon.sh. Contributed by Chris Nauroth. (cnauroth: rev 93b770f7e778835a9dd76854b435c5250835d1a8)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #195 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/195/ ) HADOOP-11966 . Variable cygwin is undefined in hadoop-config.sh when executed through hadoop-daemon.sh. Contributed by Chris Nauroth. (cnauroth: rev 93b770f7e778835a9dd76854b435c5250835d1a8) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2125 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2125/)
          HADOOP-11966. Variable cygwin is undefined in hadoop-config.sh when executed through hadoop-daemon.sh. Contributed by Chris Nauroth. (cnauroth: rev 93b770f7e778835a9dd76854b435c5250835d1a8)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2125 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2125/ ) HADOOP-11966 . Variable cygwin is undefined in hadoop-config.sh when executed through hadoop-daemon.sh. Contributed by Chris Nauroth. (cnauroth: rev 93b770f7e778835a9dd76854b435c5250835d1a8) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #185 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/185/)
          HADOOP-11966. Variable cygwin is undefined in hadoop-config.sh when executed through hadoop-daemon.sh. Contributed by Chris Nauroth. (cnauroth: rev 93b770f7e778835a9dd76854b435c5250835d1a8)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #185 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/185/ ) HADOOP-11966 . Variable cygwin is undefined in hadoop-config.sh when executed through hadoop-daemon.sh. Contributed by Chris Nauroth. (cnauroth: rev 93b770f7e778835a9dd76854b435c5250835d1a8) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2143 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2143/)
          HADOOP-11966. Variable cygwin is undefined in hadoop-config.sh when executed through hadoop-daemon.sh. Contributed by Chris Nauroth. (cnauroth: rev 93b770f7e778835a9dd76854b435c5250835d1a8)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2143 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2143/ ) HADOOP-11966 . Variable cygwin is undefined in hadoop-config.sh when executed through hadoop-daemon.sh. Contributed by Chris Nauroth. (cnauroth: rev 93b770f7e778835a9dd76854b435c5250835d1a8) hadoop-common-project/hadoop-common/CHANGES.txt

            People

            • Assignee:
              cnauroth Chris Nauroth
              Reporter:
              cnauroth Chris Nauroth
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development