Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.0.3
    • Fix Version/s: 1.2.0
    • Component/s: bin
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      when the default value for HADOOP_SLAVES is changed in hadoop-env.sh the hdfs starting (with start-dfs.sh) creates secondary namenodes on all the machines in the file conf/slaves instead of conf/masters.

      1. patch_slaves.sh_hadoop-1.0.3_fromtar.patch
        0.4 kB
        giovanni delussu
      2. patch_hadoop-config.sh_slaves.sh_trunk.patch
        2 kB
        giovanni delussu
      3. patch_hadoop-config.sh_slaves.sh_trunk_no_prefix.patch
        2 kB
        giovanni delussu
      4. patch_hadoop-config.sh_slaves.sh_branch1.patch
        1 kB
        giovanni delussu
      5. patch_hadoop-config.sh_slaves.sh_branch1_no_prefix.patch
        1 kB
        giovanni delussu
      6. patch_hadoop-config.sh_hadoop-1.0.3_fromtar.patch
        0.2 kB
        giovanni delussu

        Activity

        Hide
        giovanni delussu added a comment -

        patch for tar version hadoop-1.0.3.tar.gz
        file patched are:
        hadoop-config.sh
        slaves.sh

        Show
        giovanni delussu added a comment - patch for tar version hadoop-1.0.3.tar.gz file patched are: hadoop-config.sh slaves.sh
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12543859/patch_slaves.sh_hadoop-1.0.3_fromtar.patch
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1405//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12543859/patch_slaves.sh_hadoop-1.0.3_fromtar.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1405//console This message is automatically generated.
        Hide
        Arpit Gupta added a comment -

        Patch does not seem to be correctly formatted. For details on take a look at http://wiki.apache.org/hadoop/HowToContribute or if you use git then http://wiki.apache.org/hadoop/GitAndHadoop

        A few comments on your patch.

        Rather than assigning HOSTLIST after hadoop-env.sh is sourced we should do the following

        1. in hadoop-config.sh source the hadoop-env.sh before additional parameters are processed such as --hosts. This way if you specify the hosts at command line they will get set appropriately. This will help as in start-dfs.sh the secondary namenode is being started as

        "$bin"/hadoop-daemons.sh --config $HADOOP_CONF_DIR --hosts masters start secondarynamenode
        

        2. in slaves.sh remove the source of hadoop-env.sh as it has already been sourced by hadoop-config.sh. Thus the assignment of HADOOP_SLAVES done by hadoop-config.sh will be available rather than being overwritten by hadoop-env.sh

        Show
        Arpit Gupta added a comment - Patch does not seem to be correctly formatted. For details on take a look at http://wiki.apache.org/hadoop/HowToContribute or if you use git then http://wiki.apache.org/hadoop/GitAndHadoop A few comments on your patch. Rather than assigning HOSTLIST after hadoop-env.sh is sourced we should do the following 1. in hadoop-config.sh source the hadoop-env.sh before additional parameters are processed such as --hosts. This way if you specify the hosts at command line they will get set appropriately. This will help as in start-dfs.sh the secondary namenode is being started as "$bin" /hadoop-daemons.sh --config $HADOOP_CONF_DIR --hosts masters start secondarynamenode 2. in slaves.sh remove the source of hadoop-env.sh as it has already been sourced by hadoop-config.sh. Thus the assignment of HADOOP_SLAVES done by hadoop-config.sh will be available rather than being overwritten by hadoop-env.sh
        Hide
        giovanni delussu added a comment -

        About the format: I cloned with git the trunk repository but the two files I'm proposing to change are different. So this is not a patch for the trunk. This is only a patch for the hadoop-1.0.3.tar.gz version. What should I do? Create a git with hadoop-1.0.3.tar.gz and then after the change publish the "git diff"?

        about the comments: 1 is already the way I modified hadoop-config.sh
        2 I thought in slaves.sh the sourcing of hadoop-env.sh could serve to other stuff but if this is not the case I agree that we should remove it.

        Show
        giovanni delussu added a comment - About the format: I cloned with git the trunk repository but the two files I'm proposing to change are different. So this is not a patch for the trunk. This is only a patch for the hadoop-1.0.3.tar.gz version. What should I do? Create a git with hadoop-1.0.3.tar.gz and then after the change publish the "git diff"? about the comments: 1 is already the way I modified hadoop-config.sh 2 I thought in slaves.sh the sourcing of hadoop-env.sh could serve to other stuff but if this is not the case I agree that we should remove it.
        Hide
        Arpit Gupta added a comment -

        @Giovanni

        I checked trunk and it looks like the same problem exists in trunk where we source the hadoop-env.sh after reading the command line arguments. So the same fixes should apply there as well. Could you generate a patch for trunk also.

        For hadoop 1.0.3 we need to create the patch for branch-1 and that will be ported to branch-1.1 and branch-1.0 so the future 1.x releases can pick up the fix.

        Show
        Arpit Gupta added a comment - @Giovanni I checked trunk and it looks like the same problem exists in trunk where we source the hadoop-env.sh after reading the command line arguments. So the same fixes should apply there as well. Could you generate a patch for trunk also. For hadoop 1.0.3 we need to create the patch for branch-1 and that will be ported to branch-1.1 and branch-1.0 so the future 1.x releases can pick up the fix.
        Hide
        giovanni delussu added a comment -

        patch of the secondary namenode issue for the trunk

        Show
        giovanni delussu added a comment - patch of the secondary namenode issue for the trunk
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12544187/patch_hadoop-config.sh_slaves.sh_trunk.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

        org.apache.hadoop.ha.TestZKFailoverController

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1422//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1422//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12544187/patch_hadoop-config.sh_slaves.sh_trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverController +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1422//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1422//console This message is automatically generated.
        Hide
        giovanni delussu added a comment -

        Arpit Gupta I checked again and found that in branch-1 I need to move the setting of the HOSTLIST variable because when slaves.sh calls hadoop-config.sh it hasn't the --hosts attached so hadoop-config.sh sets HADOOP_SLAVES (by calling hadoop-env.sh) to the slaves file instead of the masters.

        Show
        giovanni delussu added a comment - Arpit Gupta I checked again and found that in branch-1 I need to move the setting of the HOSTLIST variable because when slaves.sh calls hadoop-config.sh it hasn't the --hosts attached so hadoop-config.sh sets HADOOP_SLAVES (by calling hadoop-env.sh) to the slaves file instead of the masters.
        Hide
        giovanni delussu added a comment -

        patch for branch1

        Show
        giovanni delussu added a comment - patch for branch1
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12544197/patch_hadoop-config.sh_slaves.sh_branch1.patch
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1423//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12544197/patch_hadoop-config.sh_slaves.sh_branch1.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1423//console This message is automatically generated.
        Hide
        Arpit Gupta added a comment -

        could you generate the patch using git --no-prefix, otherwiese i have to provide which file to patch.

        Also for the patches the naming convention is something like HADOOP-8767.patch for trunk and HADOOP-8767.branch-1.patch for branch-1

        Also good catch on the branch-1 changes.

        Show
        Arpit Gupta added a comment - could you generate the patch using git --no-prefix, otherwiese i have to provide which file to patch. Also for the patches the naming convention is something like HADOOP-8767 .patch for trunk and HADOOP-8767 .branch-1.patch for branch-1 Also good catch on the branch-1 changes.
        Hide
        Andy Isaacson added a comment -

        could you generate the patch using git --no-prefix, otherwiese i have to provide which file to patch.

        You can apply this patch with patch -p1.

        diff --git a/bin/hadoop-config.sh b/bin/hadoop-config.sh
        index c23cbcf..073a5fe 100644
        --- a/bin/hadoop-config.sh
        +++ b/bin/hadoop-config.sh
        % cd branch-1
        % patch -p1 < /tmp/patch_hadoop-config.sh_slaves.sh_branch1.patch
        patching file bin/hadoop-config.sh
        patching file bin/slaves.sh
        %
        

        Using --no-prefix makes the patch less useful to others.

        Show
        Andy Isaacson added a comment - could you generate the patch using git --no-prefix, otherwiese i have to provide which file to patch. You can apply this patch with patch -p1 . diff --git a/bin/hadoop-config.sh b/bin/hadoop-config.sh index c23cbcf..073a5fe 100644 --- a/bin/hadoop-config.sh +++ b/bin/hadoop-config.sh % cd branch-1 % patch -p1 < /tmp/patch_hadoop-config.sh_slaves.sh_branch1.patch patching file bin/hadoop-config.sh patching file bin/slaves.sh % Using --no-prefix makes the patch less useful to others.
        Hide
        Arpit Gupta added a comment -

        @Andy

        thanks for the info. However i was just following what we recommend to users on our wiki http://wiki.apache.org/hadoop/GitAndHadoop

        Show
        Arpit Gupta added a comment - @Andy thanks for the info. However i was just following what we recommend to users on our wiki http://wiki.apache.org/hadoop/GitAndHadoop
        Hide
        Arpit Gupta added a comment -

        also --no-prefix only removes the "a/.. b/..." so not sure how it will be less useful.

        Show
        Arpit Gupta added a comment - also --no-prefix only removes the "a/.. b/..." so not sure how it will be less useful.
        Hide
        giovanni delussu added a comment -

        patch with --no-prefix

        Show
        giovanni delussu added a comment - patch with --no-prefix
        Hide
        giovanni delussu added a comment -

        ops. plural=> patches with --no-prefix flag

        Show
        giovanni delussu added a comment - ops. plural=> patches with --no-prefix flag
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12544601/patch_hadoop-config.sh_slaves.sh_trunk_no_prefix.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1435//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1435//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12544601/patch_hadoop-config.sh_slaves.sh_trunk_no_prefix.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1435//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1435//console This message is automatically generated.
        Hide
        Arpit Gupta added a comment -

        +1 the latest changes look good to go.

        Show
        Arpit Gupta added a comment - +1 the latest changes look good to go.
        Hide
        Suresh Srinivas added a comment -

        Giovanni, I added you as a Hadoop Common contributor and assigned the jira to you.

        Show
        Suresh Srinivas added a comment - Giovanni, I added you as a Hadoop Common contributor and assigned the jira to you.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2718 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2718/)
        HADOOP-8767. Secondary namenode is started on slave nodes instead of master nodes. Contributed by Giovanni Delussu. (Revision 1383560)

        Result = SUCCESS
        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1383560
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/slaves.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2718 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2718/ ) HADOOP-8767 . Secondary namenode is started on slave nodes instead of master nodes. Contributed by Giovanni Delussu. (Revision 1383560) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1383560 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/slaves.sh
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2781 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2781/)
        HADOOP-8767. Secondary namenode is started on slave nodes instead of master nodes. Contributed by Giovanni Delussu. (Revision 1383560)

        Result = SUCCESS
        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1383560
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/slaves.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2781 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2781/ ) HADOOP-8767 . Secondary namenode is started on slave nodes instead of master nodes. Contributed by Giovanni Delussu. (Revision 1383560) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1383560 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/slaves.sh
        Hide
        Suresh Srinivas added a comment -

        +1 for the patch. Thank you Giovanni for reporting and fixing the issue. Thank you Arpit for the review.

        Show
        Suresh Srinivas added a comment - +1 for the patch. Thank you Giovanni for reporting and fixing the issue. Thank you Arpit for the review.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2742 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2742/)
        HADOOP-8767. Secondary namenode is started on slave nodes instead of master nodes. Contributed by Giovanni Delussu. (Revision 1383560)

        Result = FAILURE
        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1383560
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/slaves.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2742 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2742/ ) HADOOP-8767 . Secondary namenode is started on slave nodes instead of master nodes. Contributed by Giovanni Delussu. (Revision 1383560) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1383560 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/slaves.sh
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1163 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1163/)
        HADOOP-8767. Secondary namenode is started on slave nodes instead of master nodes. Contributed by Giovanni Delussu. (Revision 1383560)

        Result = FAILURE
        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1383560
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/slaves.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1163 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1163/ ) HADOOP-8767 . Secondary namenode is started on slave nodes instead of master nodes. Contributed by Giovanni Delussu. (Revision 1383560) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1383560 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/slaves.sh
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1194 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1194/)
        HADOOP-8767. Secondary namenode is started on slave nodes instead of master nodes. Contributed by Giovanni Delussu. (Revision 1383560)

        Result = SUCCESS
        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1383560
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/slaves.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1194 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1194/ ) HADOOP-8767 . Secondary namenode is started on slave nodes instead of master nodes. Contributed by Giovanni Delussu. (Revision 1383560) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1383560 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/slaves.sh
        Hide
        Matt Foley added a comment -

        Closed upon release of Hadoop 1.2.0.

        Show
        Matt Foley added a comment - Closed upon release of Hadoop 1.2.0.

          People

          • Assignee:
            giovanni delussu
            Reporter:
            giovanni delussu
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development