Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11668

hadoop-daemons.sh bw compat broke with --slaves change

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: scripts
    • Labels:
      None

      Description

      After introduction of "--slaves" option for the scripts, start-dfs.sh and stop-dfs.sh will no longer work in HA mode.

      This is due to multiple hostnames passed for '--hostnames' delimited with space.

      These hostnames are treated as commands and script fails.

      So, instead of delimiting with space, delimiting with comma(,) before passing to hadoop-daemons.sh will solve the problem.

      1. HADOOP-11668-01.patch
        6 kB
        Vinayakumar B
      2. HADOOP-11668-02.patch
        2 kB
        Allen Wittenauer
      3. HADOOP-11668-03.patch
        2 kB
        Allen Wittenauer
      4. HADOOP-11668-04.patch
        3 kB
        Vinayakumar B
      5. HADOOP-11668-05.patch
        3 kB
        Vinayakumar B
      6. HADOOP-11668-06.patch
        3 kB
        Vinayakumar B

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2078 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2078/)
          HADOOP-11668. hadoop-daemons.sh bw compat broke with --slaves change (Vinayakumar B via aw) (aw: rev 7711049837d69d0eeabad27f2e30fab606a4adc2)

          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemons.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/bin/yarn-daemons.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2078 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2078/ ) HADOOP-11668 . hadoop-daemons.sh bw compat broke with --slaves change (Vinayakumar B via aw) (aw: rev 7711049837d69d0eeabad27f2e30fab606a4adc2) hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemons.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/CHANGES.txt hadoop-yarn-project/hadoop-yarn/bin/yarn-daemons.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #128 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/128/)
          HADOOP-11668. hadoop-daemons.sh bw compat broke with --slaves change (Vinayakumar B via aw) (aw: rev 7711049837d69d0eeabad27f2e30fab606a4adc2)

          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemons.sh
          • hadoop-yarn-project/hadoop-yarn/bin/yarn-daemons.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #128 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/128/ ) HADOOP-11668 . hadoop-daemons.sh bw compat broke with --slaves change (Vinayakumar B via aw) (aw: rev 7711049837d69d0eeabad27f2e30fab606a4adc2) hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemons.sh hadoop-yarn-project/hadoop-yarn/bin/yarn-daemons.sh hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #119 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/119/)
          HADOOP-11668. hadoop-daemons.sh bw compat broke with --slaves change (Vinayakumar B via aw) (aw: rev 7711049837d69d0eeabad27f2e30fab606a4adc2)

          • hadoop-yarn-project/hadoop-yarn/bin/yarn-daemons.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemons.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #119 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/119/ ) HADOOP-11668 . hadoop-daemons.sh bw compat broke with --slaves change (Vinayakumar B via aw) (aw: rev 7711049837d69d0eeabad27f2e30fab606a4adc2) hadoop-yarn-project/hadoop-yarn/bin/yarn-daemons.sh hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemons.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2060 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2060/)
          HADOOP-11668. hadoop-daemons.sh bw compat broke with --slaves change (Vinayakumar B via aw) (aw: rev 7711049837d69d0eeabad27f2e30fab606a4adc2)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemons.sh
          • hadoop-yarn-project/hadoop-yarn/bin/yarn-daemons.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2060 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2060/ ) HADOOP-11668 . hadoop-daemons.sh bw compat broke with --slaves change (Vinayakumar B via aw) (aw: rev 7711049837d69d0eeabad27f2e30fab606a4adc2) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemons.sh hadoop-yarn-project/hadoop-yarn/bin/yarn-daemons.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #862 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/862/)
          HADOOP-11668. hadoop-daemons.sh bw compat broke with --slaves change (Vinayakumar B via aw) (aw: rev 7711049837d69d0eeabad27f2e30fab606a4adc2)

          • hadoop-yarn-project/hadoop-yarn/bin/yarn-daemons.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemons.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #862 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/862/ ) HADOOP-11668 . hadoop-daemons.sh bw compat broke with --slaves change (Vinayakumar B via aw) (aw: rev 7711049837d69d0eeabad27f2e30fab606a4adc2) hadoop-yarn-project/hadoop-yarn/bin/yarn-daemons.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemons.sh hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #128 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/128/)
          HADOOP-11668. hadoop-daemons.sh bw compat broke with --slaves change (Vinayakumar B via aw) (aw: rev 7711049837d69d0eeabad27f2e30fab606a4adc2)

          • hadoop-yarn-project/hadoop-yarn/bin/yarn-daemons.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemons.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #128 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/128/ ) HADOOP-11668 . hadoop-daemons.sh bw compat broke with --slaves change (Vinayakumar B via aw) (aw: rev 7711049837d69d0eeabad27f2e30fab606a4adc2) hadoop-yarn-project/hadoop-yarn/bin/yarn-daemons.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemons.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7295 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7295/)
          HADOOP-11668. hadoop-daemons.sh bw compat broke with --slaves change (Vinayakumar B via aw) (aw: rev 7711049837d69d0eeabad27f2e30fab606a4adc2)

          • hadoop-yarn-project/hadoop-yarn/bin/yarn-daemons.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemons.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7295 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7295/ ) HADOOP-11668 . hadoop-daemons.sh bw compat broke with --slaves change (Vinayakumar B via aw) (aw: rev 7711049837d69d0eeabad27f2e30fab606a4adc2) hadoop-yarn-project/hadoop-yarn/bin/yarn-daemons.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemons.sh hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          aw Allen Wittenauer added a comment -

          +1 committing to trunk.

          Thanks!

          Show
          aw Allen Wittenauer added a comment - +1 committing to trunk. Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12703332/HADOOP-11668-06.patch
          against trunk revision 790aa67.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5885//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5885//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12703332/HADOOP-11668-06.patch against trunk revision 790aa67. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5885//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5885//console This message is automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Thanks Allen Wittenauer.
          Updated again.

          Show
          vinayrpet Vinayakumar B added a comment - Thanks Allen Wittenauer . Updated again.
          Hide
          aw Allen Wittenauer added a comment -

          Missed an update:

           ${hdfsscript} --slaves --daemon "${daemonmode}" "${argv[@]}"
          

          that argv should be HADOOP_USER_PARARMS in hadoop-daemon.sh

          Show
          aw Allen Wittenauer added a comment - Missed an update: ${hdfsscript} --slaves --daemon "${daemonmode}" "${argv[@]}" that argv should be HADOOP_USER_PARARMS in hadoop-daemon.sh
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12703298/HADOOP-11668-05.patch
          against trunk revision 6444349.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

          org.apache.hadoop.http.TestHttpServer

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5883//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5883//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12703298/HADOOP-11668-05.patch against trunk revision 6444349. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.http.TestHttpServer Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5883//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5883//console This message is automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Hi Allen Wittenauer, Attached the updated patch to remove '--hosts' as well.
          Please review.

          Show
          vinayrpet Vinayakumar B added a comment - Hi Allen Wittenauer , Attached the updated patch to remove '--hosts' as well. Please review.
          Hide
          aw Allen Wittenauer added a comment -

          Oh! Good catch!

          We should probably also handle the --hosts case as well. If you can update the patch to handle that, we'll get this committed!

          Thanks!

          Show
          aw Allen Wittenauer added a comment - Oh! Good catch! We should probably also handle the --hosts case as well. If you can update the patch to handle that, we'll get this committed! Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12703007/HADOOP-11668-04.patch
          against trunk revision 95bfd08.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5870//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5870//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12703007/HADOOP-11668-04.patch against trunk revision 95bfd08. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5870//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5870//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12703007/HADOOP-11668-04.patch
          against trunk revision 95bfd08.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The following test timeouts occurred in hadoop-common-project/hadoop-common:

          org.apache.hadoop.http.TestHttpServerLifecycle

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5868//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5868//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12703007/HADOOP-11668-04.patch against trunk revision 95bfd08. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The following test timeouts occurred in hadoop-common-project/hadoop-common: org.apache.hadoop.http.TestHttpServerLifecycle Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5868//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5868//console This message is automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Thanks Allen Wittenauer for the detailed explanation.
          Latest updated patch fixes the argument parsing problem.
          Still didnt fix the end-to-end problem. Still starting of Nodes fails.

          Similar treatment required in hadoop_common_slave_mode_execute to remove --slaves, --hostnames and its values.

          Attaching the patch for the same.

          Show
          vinayrpet Vinayakumar B added a comment - Thanks Allen Wittenauer for the detailed explanation. Latest updated patch fixes the argument parsing problem. Still didnt fix the end-to-end problem. Still starting of Nodes fails. Similar treatment required in hadoop_common_slave_mode_execute to remove --slaves, --hostnames and its values. Attaching the patch for the same.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12702828/HADOOP-11668-03.patch
          against trunk revision 1b67209.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5858//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5858//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12702828/HADOOP-11668-03.patch against trunk revision 1b67209. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5858//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5858//console This message is automatically generated.
          Hide
          aw Allen Wittenauer added a comment - - edited

          -03:

          • just switch to a loop entirely. see below....

          IMO, best way to solve this is by making hostnames delimited by (,)

          Nope, definitely not.

          The start of the problems are definitely here:

          argv=(${HADOOP_USER_PARAMS[@]/start})
          

          This construction has two key issues:

          Without quotes, the array of HADOOP_USER_PARAMS will always have its metachars expanded. This means that an array of 4 elements will now become 4+n elements, depending upon what else is in there. So if a user passes:

          --config "my cool dir" 
          

          elements 2 3 4 just got expanded into my, cool, and dir rather than just the single "my cool dir".

          So if we change the construction to

          argv=("${HADOOP_USER_PARAMS[@]/start}")
          

          That element expansion no longer happens. But now we've introduced a new problem. We're doing a substitution, but turning those elements empty! This is where the empty parameter problem comes in, because this means that if we had:

          hadoop-daemons.sh --hostnames "1 2" start namenode
          

          We'd end up with:

          argv[0]=--hostnames
          argv[1]="1 2"
          argv[2]=""
          argv[3]="namenode"

          after the substitution.

          Then when we get to

          "${argv[@]}" 
          

          on the exec line, it turns into:

          hdfs --slaves --daemon "start" --hostnames "1 2" "" namenode

          Thus we also need to filter out this empty element array.

          So why don't we just switch to using commas here? Because as evidenced by the above, it doesn't actually fix all the problems with metachar expansion. If any other parameter has them, it's going to blow up in our face. The other problem we've got is backward compatibility. A lot of people use hadoop/yarn-daemons.sh in scripts, and changing this to use commas would be a pretty hefty tax especially when we know we can fix it another way.

          One of the goals I had in mind with this code was to avoid a loop. But there's still another problem here:

          if a hostname has start, stop, or status, it's going to get removed. Since we already have the loop now to deal with the empty element, we might as well fix that bug too by using a loop rather than cheating. We still have a problem if some other param is specifically start/stop/status (e.g., --config start), but there's not much we can do about that without building a pretty complex test for what mode we're in.

          Show
          aw Allen Wittenauer added a comment - - edited -03: just switch to a loop entirely. see below.... IMO, best way to solve this is by making hostnames delimited by (,) Nope, definitely not. The start of the problems are definitely here: argv=(${HADOOP_USER_PARAMS[@]/start}) This construction has two key issues: Without quotes, the array of HADOOP_USER_PARAMS will always have its metachars expanded. This means that an array of 4 elements will now become 4+n elements, depending upon what else is in there. So if a user passes: --config "my cool dir" elements 2 3 4 just got expanded into my, cool, and dir rather than just the single "my cool dir". So if we change the construction to argv=( "${HADOOP_USER_PARAMS[@]/start}" ) That element expansion no longer happens. But now we've introduced a new problem. We're doing a substitution, but turning those elements empty! This is where the empty parameter problem comes in, because this means that if we had: hadoop-daemons.sh --hostnames "1 2" start namenode We'd end up with: argv [0] =--hostnames argv [1] ="1 2" argv [2] ="" argv [3] ="namenode" after the substitution. Then when we get to "${argv[@]}" on the exec line, it turns into: hdfs --slaves --daemon "start" --hostnames "1 2" "" namenode Thus we also need to filter out this empty element array. So why don't we just switch to using commas here? Because as evidenced by the above, it doesn't actually fix all the problems with metachar expansion. If any other parameter has them, it's going to blow up in our face. The other problem we've got is backward compatibility. A lot of people use hadoop/yarn-daemons.sh in scripts, and changing this to use commas would be a pretty hefty tax especially when we know we can fix it another way. One of the goals I had in mind with this code was to avoid a loop. But there's still another problem here: if a hostname has start, stop, or status, it's going to get removed. Since we already have the loop now to deal with the empty element, we might as well fix that bug too by using a loop rather than cheating. We still have a problem if some other param is specifically start/stop/status (e.g., --config start), but there's not much we can do about that without building a pretty complex test for what mode we're in.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Thanks Allen Wittenauer.

          The problem was two fold:
          We were not preserving quotes around parameters that contained $IFS due to lack of quoting around the array deletion
          The then deleted array elements were retained and show up as an empty argument.

          I don't think the problem is having empty arguments, I have verified in my case, there were no empty arguments.
          In my case
          1. start-dfs.sh will call hadoop-daemons.sh as below
          hadoop-daemons.sh --config /home/vinay/install/conf --hostnames 'server3 server1' start namenode
          2. hadoop-daemons.sh passes these args to hdfscript as below
          hdfs --slaves --daemon start --config /home/vinay/install/conf --hostnames server3 server1 namenode

          So the actual problem found is, in Original HADOOP_USER_PARAMS array, 'server3 server1' will be one element. But after this

          argv=(${HADOOP_USER_PARAMS[@]/start})

          it becomes two separate elements in argv. Even when original HADOOP_USER_PARAMS printed as

          ${HADOOP_USER_PARAMS [@]}

          it shows without quotes, which means separate arguments to hdfsscript.

          IMO, best way to solve this is by making hostnames delimited by (,)

          Any thoughts?

          Show
          vinayrpet Vinayakumar B added a comment - Thanks Allen Wittenauer . The problem was two fold: We were not preserving quotes around parameters that contained $IFS due to lack of quoting around the array deletion The then deleted array elements were retained and show up as an empty argument. I don't think the problem is having empty arguments, I have verified in my case, there were no empty arguments. In my case 1. start-dfs.sh will call hadoop-daemons.sh as below hadoop-daemons.sh --config /home/vinay/install/conf --hostnames 'server3 server1' start namenode 2. hadoop-daemons.sh passes these args to hdfscript as below hdfs --slaves --daemon start --config /home/vinay/install/conf --hostnames server3 server1 namenode So the actual problem found is, in Original HADOOP_USER_PARAMS array, 'server3 server1' will be one element. But after this argv=(${HADOOP_USER_PARAMS[@]/start}) it becomes two separate elements in argv . Even when original HADOOP_USER_PARAMS printed as ${HADOOP_USER_PARAMS [@]} it shows without quotes, which means separate arguments to hdfsscript. IMO, best way to solve this is by making hostnames delimited by (,) Any thoughts?
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12702627/HADOOP-11668-02.patch
          against trunk revision ed70fa1.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5849//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5849//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12702627/HADOOP-11668-02.patch against trunk revision ed70fa1. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5849//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5849//console This message is automatically generated.
          Hide
          aw Allen Wittenauer added a comment - - edited

          -02:

          • This fixes hadoop-daemons.sh and yarn-daemons.sh so that they work with multiple hosts.

          The problem was two fold:

          • We were not preserving quotes around parameters that contained $IFS due to lack of quoting around the array deletion
          • The then deleted array elements were retained and show up as an empty argument.
          Show
          aw Allen Wittenauer added a comment - - edited -02: This fixes hadoop-daemons.sh and yarn-daemons.sh so that they work with multiple hosts. The problem was two fold: We were not preserving quotes around parameters that contained $IFS due to lack of quoting around the array deletion The then deleted array elements were retained and show up as an empty argument.
          Hide
          aw Allen Wittenauer added a comment -

          Re-opening. The problem here isn't start/stop, it's *-daemons.sh, which are now broken.

          Show
          aw Allen Wittenauer added a comment - Re-opening. The problem here isn't start/stop, it's *-daemons.sh, which are now broken.
          Hide
          aw Allen Wittenauer added a comment -

          Closing this in favor of HADOOP-11590, which rewrites these scripts.

          Show
          aw Allen Wittenauer added a comment - Closing this in favor of HADOOP-11590 , which rewrites these scripts.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12702417/HADOOP-11668-01.patch
          against trunk revision 3560180.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The following test timeouts occurred in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.conf.TestConfigurationDeprecation
          org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5841//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5841//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12702417/HADOOP-11668-01.patch against trunk revision 3560180. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The following test timeouts occurred in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.conf.TestConfigurationDeprecation org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5841//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5841//console This message is automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Attaching the patch.

          Delimited hostnames with comma(,)
          Using the latest 'hdfs --slaves' script in start-dfs.sh and stop-dfs.sh

          Show
          vinayrpet Vinayakumar B added a comment - Attaching the patch. Delimited hostnames with comma(,) Using the latest 'hdfs --slaves' script in start-dfs.sh and stop-dfs.sh

            People

            • Assignee:
              vinayrpet Vinayakumar B
              Reporter:
              vinayrpet Vinayakumar B
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development