Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: scripts
    • Labels:
    • Target Version/s:

      Description

      In order to enable significantly better unit testing as well as enhanced functionality, large portions of *-config.sh should be pulled into functions. See first comment for more.

      1. HADOOP-12294.01.patch
        24 kB
        Allen Wittenauer
      2. HADOOP-12249.00.patch
        23 kB
        Allen Wittenauer

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #271 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/271/)
          HADOOP-12249. pull argument parsing into a function (aw) (aw: rev 666cafca8d3c928f3470a03ae9dedb27e27f8f0e)

          • hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh
          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-balancer.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh
          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-balancer.sh
          • hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh
          • hadoop-yarn-project/hadoop-yarn/bin/yarn
          • hadoop-common-project/hadoop-common/src/main/bin/slaves.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          • hadoop-mapreduce-project/bin/mapred
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #271 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/271/ ) HADOOP-12249 . pull argument parsing into a function (aw) (aw: rev 666cafca8d3c928f3470a03ae9dedb27e27f8f0e) hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-balancer.sh hadoop-common-project/hadoop-common/CHANGES.txt hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-balancer.sh hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh hadoop-yarn-project/hadoop-yarn/bin/yarn hadoop-common-project/hadoop-common/src/main/bin/slaves.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs hadoop-mapreduce-project/bin/mapred hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #263 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/263/)
          HADOOP-12249. pull argument parsing into a function (aw) (aw: rev 666cafca8d3c928f3470a03ae9dedb27e27f8f0e)

          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh
          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-balancer.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop
          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-balancer.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          • hadoop-mapreduce-project/bin/mapred
          • hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-common-project/hadoop-common/src/main/bin/slaves.sh
          • hadoop-yarn-project/hadoop-yarn/bin/yarn
          • hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #263 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/263/ ) HADOOP-12249 . pull argument parsing into a function (aw) (aw: rev 666cafca8d3c928f3470a03ae9dedb27e27f8f0e) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-balancer.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-balancer.sh hadoop-common-project/hadoop-common/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs hadoop-mapreduce-project/bin/mapred hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-common-project/hadoop-common/src/main/bin/slaves.sh hadoop-yarn-project/hadoop-yarn/bin/yarn hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2201 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2201/)
          HADOOP-12249. pull argument parsing into a function (aw) (aw: rev 666cafca8d3c928f3470a03ae9dedb27e27f8f0e)

          • hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh
          • hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh
          • hadoop-common-project/hadoop-common/src/main/bin/slaves.sh
          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-balancer.sh
          • hadoop-mapreduce-project/bin/mapred
          • hadoop-yarn-project/hadoop-yarn/bin/yarn
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh
          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-balancer.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2201 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2201/ ) HADOOP-12249 . pull argument parsing into a function (aw) (aw: rev 666cafca8d3c928f3470a03ae9dedb27e27f8f0e) hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh hadoop-common-project/hadoop-common/src/main/bin/slaves.sh hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-balancer.sh hadoop-mapreduce-project/bin/mapred hadoop-yarn-project/hadoop-yarn/bin/yarn hadoop-common-project/hadoop-common/CHANGES.txt hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-balancer.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2220 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2220/)
          HADOOP-12249. pull argument parsing into a function (aw) (aw: rev 666cafca8d3c928f3470a03ae9dedb27e27f8f0e)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-balancer.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-common-project/hadoop-common/src/main/bin/slaves.sh
          • hadoop-yarn-project/hadoop-yarn/bin/yarn
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-balancer.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh
          • hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh
          • hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh
          • hadoop-mapreduce-project/bin/mapred
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2220 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2220/ ) HADOOP-12249 . pull argument parsing into a function (aw) (aw: rev 666cafca8d3c928f3470a03ae9dedb27e27f8f0e) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-balancer.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-common-project/hadoop-common/src/main/bin/slaves.sh hadoop-yarn-project/hadoop-yarn/bin/yarn hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-balancer.sh hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh hadoop-mapreduce-project/bin/mapred
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #274 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/274/)
          HADOOP-12249. pull argument parsing into a function (aw) (aw: rev 666cafca8d3c928f3470a03ae9dedb27e27f8f0e)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh
          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-balancer.sh
          • hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-mapreduce-project/bin/mapred
          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-balancer.sh
          • hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh
          • hadoop-yarn-project/hadoop-yarn/bin/yarn
          • hadoop-common-project/hadoop-common/src/main/bin/slaves.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop
          • hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #274 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/274/ ) HADOOP-12249 . pull argument parsing into a function (aw) (aw: rev 666cafca8d3c928f3470a03ae9dedb27e27f8f0e) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-balancer.sh hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-mapreduce-project/bin/mapred hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-balancer.sh hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh hadoop-yarn-project/hadoop-yarn/bin/yarn hadoop-common-project/hadoop-common/src/main/bin/slaves.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1004 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1004/)
          HADOOP-12249. pull argument parsing into a function (aw) (aw: rev 666cafca8d3c928f3470a03ae9dedb27e27f8f0e)

          • hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh
          • hadoop-common-project/hadoop-common/src/main/bin/slaves.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh
          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh
          • hadoop-yarn-project/hadoop-yarn/bin/yarn
          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-balancer.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop
          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-balancer.sh
          • hadoop-mapreduce-project/bin/mapred
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1004 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1004/ ) HADOOP-12249 . pull argument parsing into a function (aw) (aw: rev 666cafca8d3c928f3470a03ae9dedb27e27f8f0e) hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh hadoop-common-project/hadoop-common/src/main/bin/slaves.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh hadoop-yarn-project/hadoop-yarn/bin/yarn hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-balancer.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-balancer.sh hadoop-mapreduce-project/bin/mapred
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8253 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8253/)
          HADOOP-12249. pull argument parsing into a function (aw) (aw: rev 666cafca8d3c928f3470a03ae9dedb27e27f8f0e)

          • hadoop-common-project/hadoop-common/src/main/bin/slaves.sh
          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-balancer.sh
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh
          • hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-mapreduce-project/bin/mapred
          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-balancer.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop
          • hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/bin/yarn
          • hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8253 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8253/ ) HADOOP-12249 . pull argument parsing into a function (aw) (aw: rev 666cafca8d3c928f3470a03ae9dedb27e27f8f0e) hadoop-common-project/hadoop-common/src/main/bin/slaves.sh hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-balancer.sh hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-mapreduce-project/bin/mapred hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-balancer.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh hadoop-common-project/hadoop-common/CHANGES.txt hadoop-yarn-project/hadoop-yarn/bin/yarn hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          Hide
          aw Allen Wittenauer added a comment -

          Thanks!

          Committed to trunk.

          Show
          aw Allen Wittenauer added a comment - Thanks! Committed to trunk.
          Hide
          owen.omalley Owen O'Malley added a comment -

          This looks good to me, Allen. +1

          Show
          owen.omalley Owen O'Malley added a comment - This looks good to me, Allen. +1
          Hide
          busbey Sean Busbey added a comment -

          that makes sense. Thanks for the explanation.

          Show
          busbey Sean Busbey added a comment - that makes sense. Thanks for the explanation.
          Hide
          aw Allen Wittenauer added a comment -

          Is there a compatibility reason we need MYNAME to exist rather than e.g. using BASH_SOURCE directly?

          The ${BASH_SOURCE-$0} "math" is needed in a variety of places on a per executable basis. One of the big ways it is used is to determine where the libexec directory is located at if the user doesn't have any env vars set. So it's shoved in a variable for convenience. In older scripts, it's generally used as "$this"

          Show
          aw Allen Wittenauer added a comment - Is there a compatibility reason we need MYNAME to exist rather than e.g. using BASH_SOURCE directly? The ${BASH_SOURCE-$0} "math" is needed in a variety of places on a per executable basis. One of the big ways it is used is to determine where the libexec directory is located at if the user doesn't have any env vars set. So it's shoved in a variable for convenience. In older scripts, it's generally used as "$this"
          Hide
          busbey Sean Busbey added a comment -

          nit:

          +MYNAME="${BASH_SOURCE-$0}"
          +
          

          we repeat this phrase a lot. Is there a compatibility reason we need MYNAME to exist rather than e.g. using BASH_SOURCE directly?

          +1 (non-binding)

          Show
          busbey Sean Busbey added a comment - nit: +MYNAME= "${BASH_SOURCE-$0}" + we repeat this phrase a lot. Is there a compatibility reason we need MYNAME to exist rather than e.g. using BASH_SOURCE directly? +1 (non-binding)
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 40s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 37s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 shellcheck 0m 5s There were no new shellcheck (v0.3.3) issues.
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 20s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 common tests 22m 15s Tests passed in hadoop-common.
          +1 common tests 1m 38s Tests passed in hadoop-kms.
          +1 hdfs tests 161m 27s Tests passed in hadoop-hdfs.
          +1 hdfs tests 3m 38s Tests passed in hadoop-hdfs-httpfs.
              223m 21s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12748008/HADOOP-12294.01.patch
          Optional Tests javadoc javac unit shellcheck
          git revision trunk / ddc867ce
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7373/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-kms test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7373/artifact/patchprocess/testrun_hadoop-kms.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7373/artifact/patchprocess/testrun_hadoop-hdfs.txt
          hadoop-hdfs-httpfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7373/artifact/patchprocess/testrun_hadoop-hdfs-httpfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7373/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7373/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 40s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 37s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 shellcheck 0m 5s There were no new shellcheck (v0.3.3) issues. +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 20s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 common tests 22m 15s Tests passed in hadoop-common. +1 common tests 1m 38s Tests passed in hadoop-kms. +1 hdfs tests 161m 27s Tests passed in hadoop-hdfs. +1 hdfs tests 3m 38s Tests passed in hadoop-hdfs-httpfs.     223m 21s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12748008/HADOOP-12294.01.patch Optional Tests javadoc javac unit shellcheck git revision trunk / ddc867ce hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7373/artifact/patchprocess/testrun_hadoop-common.txt hadoop-kms test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7373/artifact/patchprocess/testrun_hadoop-kms.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7373/artifact/patchprocess/testrun_hadoop-hdfs.txt hadoop-hdfs-httpfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7373/artifact/patchprocess/testrun_hadoop-hdfs-httpfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7373/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7373/console This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          -01:

          • minor cleanup
          • fix a shellcheck error
          • allow usage information to be reset
          • allow for SUBCOMMAND text to be replaced (future usage)
          Show
          aw Allen Wittenauer added a comment - -01: minor cleanup fix a shellcheck error allow usage information to be reset allow for SUBCOMMAND text to be replaced (future usage)
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 52s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 43s There were no new javac warning messages.
          +1 javadoc 9m 40s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 shellcheck 0m 5s The applied patch generated 1 new shellcheck (v0.3.3) issues (total was 25, now 26).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 21s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 common tests 22m 25s Tests passed in hadoop-common.
          +1 common tests 1m 36s Tests passed in hadoop-kms.
          -1 hdfs tests 159m 36s Tests failed in hadoop-hdfs.
          +1 hdfs tests 3m 35s Tests passed in hadoop-hdfs-httpfs.
              221m 53s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestDistributedFileSystem



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12746260/HADOOP-12249.00.patch
          Optional Tests javadoc javac unit shellcheck
          git revision trunk / df1e8ce
          shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/7306/artifact/patchprocess/diffpatchshellcheck.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7306/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-kms test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7306/artifact/patchprocess/testrun_hadoop-kms.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7306/artifact/patchprocess/testrun_hadoop-hdfs.txt
          hadoop-hdfs-httpfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7306/artifact/patchprocess/testrun_hadoop-hdfs-httpfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7306/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7306/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 52s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 43s There were no new javac warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 shellcheck 0m 5s The applied patch generated 1 new shellcheck (v0.3.3) issues (total was 25, now 26). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 21s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 common tests 22m 25s Tests passed in hadoop-common. +1 common tests 1m 36s Tests passed in hadoop-kms. -1 hdfs tests 159m 36s Tests failed in hadoop-hdfs. +1 hdfs tests 3m 35s Tests passed in hadoop-hdfs-httpfs.     221m 53s   Reason Tests Failed unit tests hadoop.hdfs.TestDistributedFileSystem Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12746260/HADOOP-12249.00.patch Optional Tests javadoc javac unit shellcheck git revision trunk / df1e8ce shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/7306/artifact/patchprocess/diffpatchshellcheck.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7306/artifact/patchprocess/testrun_hadoop-common.txt hadoop-kms test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7306/artifact/patchprocess/testrun_hadoop-kms.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7306/artifact/patchprocess/testrun_hadoop-hdfs.txt hadoop-hdfs-httpfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7306/artifact/patchprocess/testrun_hadoop-hdfs-httpfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7306/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7306/console This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment - - edited

          -00:

          • hadoop-config.sh argument parsing moved to hadoop-functions hadoop_parse_args
          • reworked hadoop_usage everywhere so that main entry points can advertise exactly which options they support rather than using the generic one
          • moved actual, global options into hadoop_parse args
          • moved the column create code into a separate function so that both options and subcommands can get columnizated.
          Show
          aw Allen Wittenauer added a comment - - edited -00: hadoop-config.sh argument parsing moved to hadoop-functions hadoop_parse_args reworked hadoop_usage everywhere so that main entry points can advertise exactly which options they support rather than using the generic one moved actual, global options into hadoop_parse args moved the column create code into a separate function so that both options and subcommands can get columnizated.
          Hide
          aw Allen Wittenauer added a comment -

          The current way all of the *-config.sh code is written it assumes that all argv work must be done at the global layer. During the rewrite, this was done in order to facilitate backward compatibility with non-Hadoop components.

          However.

          This is actually worked around very easily: the only real requirement is that items get pulled off the stack via shift. This can be done by setting a global var that tells the caller of the argument parsing how many to shift off the stack after they've called our argument parser.

          In turn, this means that bin/* and sbin/* can also then be converted to use primarily functions, further enabling better unit testing.

          Show
          aw Allen Wittenauer added a comment - The current way all of the *-config.sh code is written it assumes that all argv work must be done at the global layer. During the rewrite, this was done in order to facilitate backward compatibility with non-Hadoop components. However. This is actually worked around very easily: the only real requirement is that items get pulled off the stack via shift. This can be done by setting a global var that tells the caller of the argument parsing how many to shift off the stack after they've called our argument parser. In turn, this means that bin/* and sbin/* can also then be converted to use primarily functions, further enabling better unit testing.

            People

            • Assignee:
              aw Allen Wittenauer
              Reporter:
              aw Allen Wittenauer
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development