Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4533

start-dfs.sh ignored additional parameters besides -upgrade

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha
    • Fix Version/s: 2.1.0-beta
    • Component/s: datanode, namenode
    • Labels:
    • Hadoop Flags:
      Reviewed
    • Tags:
      HDFS

      Description

      start-dfs.sh only takes -upgrade option and ignored others.

      So If run the following command, it will ignore the clusterId option.

      start-dfs.sh -upgrade -clusterId 1234

      1. HDFS-4533.patch
        0.7 kB
        Fengdong Yu
      2. HDFS-4533_2.patch
        0.5 kB
        Fengdong Yu

        Issue Links

          Activity

          Hide
          Fengdong Yu added a comment -

          Added patch to take other options in start-dfs.sh.

          Show
          Fengdong Yu added a comment - Added patch to take other options in start-dfs.sh.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12571164/HDFS-4533.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4008//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12571164/HDFS-4533.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4008//console This message is automatically generated.
          Hide
          Suresh Srinivas added a comment -

          Instead of the while loop the following code will add the remaining options, right?

          nameStartOpt="$nameStartOpts $@"
          

          Did you manually test this?

          Show
          Suresh Srinivas added a comment - Instead of the while loop the following code will add the remaining options, right? nameStartOpt= "$nameStartOpts $@" Did you manually test this?
          Hide
          Fengdong Yu added a comment -

          Hi suresh,

          I did manually test on our cluster. It does work well. I changed the patch to using $@ instead of while loop.

          I generate patch under svn: "hadoop/common/branches/branch-2.0.3-alpha/hadoop-hdfs-project/hadoop-hdfs/src/main/bin", I don't think this a wrong svn directory for my patch.

          Show
          Fengdong Yu added a comment - Hi suresh, I did manually test on our cluster. It does work well. I changed the patch to using $@ instead of while loop. I generate patch under svn: "hadoop/common/branches/branch-2.0.3-alpha/hadoop-hdfs-project/hadoop-hdfs/src/main/bin", I don't think this a wrong svn directory for my patch.
          Hide
          Fengdong Yu added a comment -

          should I generate patch under svn: "hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin" ?

          Show
          Fengdong Yu added a comment - should I generate patch under svn: "hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin" ?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12573006/HDFS-4533.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4067//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12573006/HDFS-4533.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4067//console This message is automatically generated.
          Hide
          Fengdong Yu added a comment -

          Generate patch against trunk revision.

          Show
          Fengdong Yu added a comment - Generate patch against trunk revision.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12573007/HDFS-4533.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.datanode.TestDataDirs

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4069//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4069//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12573007/HDFS-4533.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.datanode.TestDataDirs +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4069//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4069//console This message is automatically generated.
          Hide
          Colin Patrick McCabe added a comment -

          I think I would rather have something like this:

          usage="Usage: start-dfs.sh [-upgrade|-rollback] [-nnOpt <additional NameNode options>]
          

          that way you could do something like this:

          ./start-dfs.sh -nnOpt '-clusterId 123'
          

          That way, it would be easy for us to add a way to pass options to the JournalNode, DataNode, etc. in the future. Just dumping everything we can't parse into the NameNode arguments seems suboptimal... and will prevent us giving helpful error messages when the user messes up.

          Show
          Colin Patrick McCabe added a comment - I think I would rather have something like this: usage="Usage: start-dfs.sh [-upgrade|-rollback] [-nnOpt <additional NameNode options>] that way you could do something like this: ./start-dfs.sh -nnOpt '-clusterId 123' That way, it would be easy for us to add a way to pass options to the JournalNode , DataNode , etc. in the future. Just dumping everything we can't parse into the NameNode arguments seems suboptimal... and will prevent us giving helpful error messages when the user messes up.
          Hide
          Suresh Srinivas added a comment - - edited

          +1 for the patch.

          Colin, please file another jira to do what you suggested. Personally I do not think it adds value beyond what is being don now and is necessary. The current patch looks good to me.

          Show
          Suresh Srinivas added a comment - - edited +1 for the patch. Colin, please file another jira to do what you suggested. Personally I do not think it adds value beyond what is being don now and is necessary. The current patch looks good to me.
          Hide
          Colin Patrick McCabe added a comment -

          Hi Suresh,

          My concern is that this change will regress error handling.

          Before the patch:

          cmccabe@keter:/h> ./sbin/start-dfs.sh -crazygonuts
          Usage: start-dfs.sh [-upgrade|-rollback]
          

          After the patch:

          cmccabe@keter:/h> ./sbin/start-dfs.sh -crazygonuts
          [... silently does the wrong thing ...]
          

          Does that make sense?

          Show
          Colin Patrick McCabe added a comment - Hi Suresh, My concern is that this change will regress error handling. Before the patch: cmccabe@keter:/h> ./sbin/start-dfs.sh -crazygonuts Usage: start-dfs.sh [-upgrade|-rollback] After the patch: cmccabe@keter:/h> ./sbin/start-dfs.sh -crazygonuts [... silently does the wrong thing ...] Does that make sense?
          Hide
          Suresh Srinivas added a comment -

          Colin Patrick McCabe Can you explain how after the patch the invalid option is ignored? Did you run this command with the patch?

          Show
          Suresh Srinivas added a comment - Colin Patrick McCabe Can you explain how after the patch the invalid option is ignored? Did you run this command with the patch?
          Hide
          Colin Patrick McCabe added a comment -

          I ran the command with the patch, and ./sbin/start-dfs.sh -crazygonuts was rejected. However, ./sbin/start-dfs.sh -upgrade -crazygonuts was accepted without a visible error message.

          Given that the obvious error handling cases are covered, I withdraw my earlier objection. I still think we should add a way to pass arguments to the other daemons, but I suppose that can be added in a separate JIRA when we actually need it.

          Show
          Colin Patrick McCabe added a comment - I ran the command with the patch, and ./sbin/start-dfs.sh -crazygonuts was rejected. However, ./sbin/start-dfs.sh -upgrade -crazygonuts was accepted without a visible error message. Given that the obvious error handling cases are covered, I withdraw my earlier objection. I still think we should add a way to pass arguments to the other daemons, but I suppose that can be added in a separate JIRA when we actually need it.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3739 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3739/)
          HDFS-4533. start-dfs.sh ignores additional parameters besides -upgrade. Contributed by Fengdong Yu. (Revision 1481178)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1481178
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-dfs.sh
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3739 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3739/ ) HDFS-4533 . start-dfs.sh ignores additional parameters besides -upgrade. Contributed by Fengdong Yu. (Revision 1481178) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1481178 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-dfs.sh
          Hide
          Suresh Srinivas added a comment -

          I committed the patch to trunk and branch-2. Thank you Fengdong!

          Show
          Suresh Srinivas added a comment - I committed the patch to trunk and branch-2. Thank you Fengdong!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #206 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/206/)
          HDFS-4533. start-dfs.sh ignores additional parameters besides -upgrade. Contributed by Fengdong Yu. (Revision 1481178)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1481178
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-dfs.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #206 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/206/ ) HDFS-4533 . start-dfs.sh ignores additional parameters besides -upgrade. Contributed by Fengdong Yu. (Revision 1481178) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1481178 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-dfs.sh
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1395 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1395/)
          HDFS-4533. start-dfs.sh ignores additional parameters besides -upgrade. Contributed by Fengdong Yu. (Revision 1481178)

          Result = FAILURE
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1481178
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-dfs.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1395 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1395/ ) HDFS-4533 . start-dfs.sh ignores additional parameters besides -upgrade. Contributed by Fengdong Yu. (Revision 1481178) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1481178 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-dfs.sh
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1422 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1422/)
          HDFS-4533. start-dfs.sh ignores additional parameters besides -upgrade. Contributed by Fengdong Yu. (Revision 1481178)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1481178
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-dfs.sh
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1422 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1422/ ) HDFS-4533 . start-dfs.sh ignores additional parameters besides -upgrade. Contributed by Fengdong Yu. (Revision 1481178) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1481178 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-dfs.sh
          Hide
          Suresh Srinivas added a comment -

          Fengdong Yu Can you please file icla - http://www.apache.org/licenses/icla.txt. Once you do that, I will add you as a contributor and assign this jira to you.

          Show
          Suresh Srinivas added a comment - Fengdong Yu Can you please file icla - http://www.apache.org/licenses/icla.txt . Once you do that, I will add you as a contributor and assign this jira to you.
          Hide
          Fengdong Yu added a comment -

          Hi Suresh,
          I Just noticed this issue was fixed, sorry for response later. I will file icla soon. Thanks.

          Show
          Fengdong Yu added a comment - Hi Suresh, I Just noticed this issue was fixed, sorry for response later. I will file icla soon. Thanks.
          Hide
          Fengdong Yu added a comment -

          Hi Suresh,
          I've send email to submit icla, Thanks.

          Show
          Fengdong Yu added a comment - Hi Suresh, I've send email to submit icla, Thanks.
          Hide
          Fengdong Yu added a comment -

          Hi Suresh,
          My ICLA was confirmed, the following is from the Secretary e-mail.

          Dear Azuryy Yu,

          This message acknowledges receipt of your ICLA, which has been filed in the Apache Software Foundation records.

          If you have been invited as a committer, please advise the project PMC that your ICLA has been filed.

          Warm Regards,

          – Craig L Russell
          Secretary, Apache Software Foundation

          Show
          Fengdong Yu added a comment - Hi Suresh, My ICLA was confirmed, the following is from the Secretary e-mail. Dear Azuryy Yu, This message acknowledges receipt of your ICLA, which has been filed in the Apache Software Foundation records. If you have been invited as a committer, please advise the project PMC that your ICLA has been filed. Warm Regards, – Craig L Russell Secretary, Apache Software Foundation
          Hide
          Suresh Srinivas added a comment -

          Fengdong Yu I added you as HDFS contributor and assigned this jira to you. Congrats and look forward to more contributions!

          Show
          Suresh Srinivas added a comment - Fengdong Yu I added you as HDFS contributor and assigned this jira to you. Congrats and look forward to more contributions!
          Hide
          Fengdong Yu added a comment -

          Thanks, Suresh.

          Show
          Fengdong Yu added a comment - Thanks, Suresh.
          Hide
          Fengdong Yu added a comment -

          There was a bug for add addtional name node's options.

          Show
          Fengdong Yu added a comment - There was a bug for add addtional name node's options.
          Hide
          Fengdong Yu added a comment -

          I reopened this issue, because there is a minor bug, which I found during my upgrade test.

          Show
          Fengdong Yu added a comment - I reopened this issue, because there is a minor bug, which I found during my upgrade test.
          Hide
          Suresh Srinivas added a comment -

          Can you describe what the bug is? Also please open another Jira, add a link to that Jira in this Jira, instead of reopening this Jira.

          Show
          Suresh Srinivas added a comment - Can you describe what the bug is? Also please open another Jira, add a link to that Jira in this Jira, instead of reopening this Jira.
          Hide
          Fengdong Yu added a comment -

          there is a typo, I've uploaded the patch here.

           nameStartOpt="$nameStartOpts $@ 
          

          it should be:

           nameStartOpt="$nameStartOpt $@
          

          I will open a new JIRA. Thanks.

          Show
          Fengdong Yu added a comment - there is a typo, I've uploaded the patch here. nameStartOpt="$nameStartOpts $@ it should be: nameStartOpt="$nameStartOpt $@ I will open a new JIRA. Thanks.
          Hide
          Suresh Srinivas added a comment -

          I should have done a better job code reviewing this. In fact I might have been the reason for this - see https://issues.apache.org/jira/browse/HDFS-4533?focusedCommentId=13597719&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13597719

          Thanks for the follow up Fengdong Yu

          Show
          Suresh Srinivas added a comment - I should have done a better job code reviewing this. In fact I might have been the reason for this - see https://issues.apache.org/jira/browse/HDFS-4533?focusedCommentId=13597719&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13597719 Thanks for the follow up Fengdong Yu
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12588502/HDFS-4533_2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4534//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4534//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12588502/HDFS-4533_2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4534//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4534//console This message is automatically generated.

            People

            • Assignee:
              Fengdong Yu
              Reporter:
              Fengdong Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development