Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1905

Improve the usability of namenode -format

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      While setting up 0.23 version based cluster, i ran into this issue. When i issue a format namenode command, which got changed in 23, it should let the user know to how to use this command in case where complete options were not specified.

      ./hdfs namenode -format

      I get the following error msg, still its not clear what and how user should use this command.

      11/05/09 15:36:25 ERROR namenode.NameNode: java.lang.IllegalArgumentException: Format must be provided with clusterid
      at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1483)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1623)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1689)

      The usability of this command can be improved.

      1. HDFS-1905-1.patch
        2 kB
        Bharath Mundlapudi
      2. HDFS-1905-2.patch
        6 kB
        Bharath Mundlapudi

        Issue Links

          Activity

          Hide
          Todd Lipcon added a comment -

          I would agree with Doug Balog's opinion expressed on the list: for the general user, "hadoop namenode -format" should automatically generate a random cluster ID.

          What is a use case in which a cluster ID would be manually specified? These are meant to be internal-facing unique identifiers (similar to storage ID which we've had for years), right?

          Show
          Todd Lipcon added a comment - I would agree with Doug Balog's opinion expressed on the list: for the general user, "hadoop namenode -format" should automatically generate a random cluster ID. What is a use case in which a cluster ID would be manually specified? These are meant to be internal-facing unique identifiers (similar to storage ID which we've had for years), right?
          Hide
          Bharath Mundlapudi added a comment -

          Cluster ID is displayed on the dfshealth web page. If we are having multiple clusters then having a proper cluster name defined by admins will be useful.

          If user executes the following command, then correct usage is indeed displayed.
          ./hdfs namenode -format -help

          This should be corrected from all the paths.

          Show
          Bharath Mundlapudi added a comment - Cluster ID is displayed on the dfshealth web page. If we are having multiple clusters then having a proper cluster name defined by admins will be useful. If user executes the following command, then correct usage is indeed displayed. ./hdfs namenode -format -help This should be corrected from all the paths.
          Hide
          Matt Foley added a comment -

          What is a use case in which a cluster ID would be manually specified?

          I can imagine some nasty manual recovery process where you might wish to initialize a clean environment with a specified clusterId, followed by manual injection of data recovered from image and edits files. Not something I would want to do but probably should be supported.

          However, I agree "-format" without other args should create a new cid if no old cid is available. A prompt would be appropriate, same as currently done with re-use of an available old cid.

          Show
          Matt Foley added a comment - What is a use case in which a cluster ID would be manually specified? I can imagine some nasty manual recovery process where you might wish to initialize a clean environment with a specified clusterId, followed by manual injection of data recovered from image and edits files. Not something I would want to do but probably should be supported. However, I agree "-format" without other args should create a new cid if no old cid is available. A prompt would be appropriate, same as currently done with re-use of an available old cid.
          Hide
          Suresh Srinivas added a comment -

          The reason format requires a cluster ID is the following:

          1. When you add new namenodes to the existing cluster, they become the part of the federated cluster only if the same cluster ID is used. Otherwise, it is a different cluster.
          2. This leaves us two choices - allow automatic generation of cluster ID for the first namenode. Then expect admin to use the same cluster ID for formatting additional namenodes. But this leaves us with admin accidentally formatting additional namenode without specifying a cluster ID and a cluster ID is automatically is generated. The namenode that was intended to be part of the same cluster now is not!

          Given this, we decided not to automatically generate cluster ID. An admin must specify it.

          > A prompt would be appropriate, same as currently done with re-use of an available old cid.
          I do not think this solves the problem I stated.

          Show
          Suresh Srinivas added a comment - The reason format requires a cluster ID is the following: When you add new namenodes to the existing cluster, they become the part of the federated cluster only if the same cluster ID is used. Otherwise, it is a different cluster. This leaves us two choices - allow automatic generation of cluster ID for the first namenode. Then expect admin to use the same cluster ID for formatting additional namenodes. But this leaves us with admin accidentally formatting additional namenode without specifying a cluster ID and a cluster ID is automatically is generated. The namenode that was intended to be part of the same cluster now is not! Given this, we decided not to automatically generate cluster ID. An admin must specify it. > A prompt would be appropriate, same as currently done with re-use of an available old cid. I do not think this solves the problem I stated.
          Hide
          Todd Lipcon added a comment -

          Since the vast majority of users will not be using the federation feature, I think it's best to optimize for the common case and not for federated clusters. That is to say, we don't want to pollute the mental model of HDFS for new users by making them understand cluster IDs, block pools, etc.

          But this leaves us with admin accidentally formatting additional namenode without specifying a cluster ID and a cluster ID is automatically is generated. The namenode that was intended to be part of the same cluster now is not!

          Sure, but they will figure this out before they put any data into it (since the DNs won't talk to this NN). And then calling format again with the correct cluster ID specified is no problem at all for them.

          Show
          Todd Lipcon added a comment - Since the vast majority of users will not be using the federation feature, I think it's best to optimize for the common case and not for federated clusters. That is to say, we don't want to pollute the mental model of HDFS for new users by making them understand cluster IDs, block pools, etc. But this leaves us with admin accidentally formatting additional namenode without specifying a cluster ID and a cluster ID is automatically is generated. The namenode that was intended to be part of the same cluster now is not! Sure, but they will figure this out before they put any data into it (since the DNs won't talk to this NN). And then calling format again with the correct cluster ID specified is no problem at all for them.
          Hide
          Suresh Srinivas added a comment -

          Couple of other comments I missed:
          During design we wanted to ensure cluster ID is unique - to avoid accidentally naming two clusters with the same cluster ID. To do that, we have an option to generate a unique, UUID like, cluster ID.

          Instead of a complicated UUID kind of to identify a cluster, it would be good to use a name to identify a cluster. Given the small number of cluster, coming with a simple naming scheme should not be hard.

          Given that we should delete the functionality to generate cluster ID.

          Show
          Suresh Srinivas added a comment - Couple of other comments I missed: During design we wanted to ensure cluster ID is unique - to avoid accidentally naming two clusters with the same cluster ID. To do that, we have an option to generate a unique, UUID like, cluster ID. Instead of a complicated UUID kind of to identify a cluster, it would be good to use a name to identify a cluster. Given the small number of cluster, coming with a simple naming scheme should not be hard. Given that we should delete the functionality to generate cluster ID.
          Hide
          Suresh Srinivas added a comment -

          > we don't want to pollute the mental model of HDFS for new users by making them understand cluster IDs, block pools, etc.
          I disagree with you on this. Cluster ID though is being added as part of federation, I do not think pollutes the mental model.

          What is the cluster today? It is all the nodes sharing the same namespaceID, that is automatically generated and shared by all the nodes. Cluster ID makes it much cleaner where user identifiable name is shared by all the nodes and will identify all the nodes in the cluster. I am not sure if this is such a complicated idea that disrupts the HDFS model.

          Further, even without federation, we should have had such an identifier in the first place, instead of namespaceID, which happened to become cluster ID equivalent.

          Show
          Suresh Srinivas added a comment - > we don't want to pollute the mental model of HDFS for new users by making them understand cluster IDs, block pools, etc. I disagree with you on this. Cluster ID though is being added as part of federation, I do not think pollutes the mental model. What is the cluster today? It is all the nodes sharing the same namespaceID, that is automatically generated and shared by all the nodes. Cluster ID makes it much cleaner where user identifiable name is shared by all the nodes and will identify all the nodes in the cluster. I am not sure if this is such a complicated idea that disrupts the HDFS model. Further, even without federation, we should have had such an identifier in the first place, instead of namespaceID, which happened to become cluster ID equivalent.
          Hide
          Todd Lipcon added a comment -

          I agree that cluster ID is a nicer construct than namespace ID. But it doesn't replace it, since we still have the namespaceID in NNStorage.

          Perhaps a nice compromise would be the following:

          • "hadoop namenode -format" gains a required argument for cluster ID. ie "hadoop namenode -format mycluster". If you don't specify this it should print usage info.
          • "hadoop namenode -upgrade" by default will carry over the old namespaceID as the new cluster's cluster ID? Alternatively one may provide a cluster ID with "hadoop namenode -upgrade -clusterid foo"?

          Another question: if cluster ID is meant to be a user-visible "nice name" – how can one rename a cluster?

          Show
          Todd Lipcon added a comment - I agree that cluster ID is a nicer construct than namespace ID. But it doesn't replace it, since we still have the namespaceID in NNStorage. Perhaps a nice compromise would be the following: "hadoop namenode -format" gains a required argument for cluster ID. ie "hadoop namenode -format mycluster". If you don't specify this it should print usage info. "hadoop namenode -upgrade" by default will carry over the old namespaceID as the new cluster's cluster ID? Alternatively one may provide a cluster ID with "hadoop namenode -upgrade -clusterid foo"? Another question: if cluster ID is meant to be a user-visible "nice name" – how can one rename a cluster?
          Hide
          Suresh Srinivas added a comment -

          > ... "hadoop namenode -format mycluster". If you don't specify this it should print usage info.
          This is what is done now.
          > one may provide a cluster ID with "hadoop namenode -upgrade -clusterid foo
          This is what is done now.

          > if cluster ID is meant to be a user-visible "nice name" – how can one rename a cluster?
          Currently there is not functionality to do this. But we may need to add this to facilitate merging of clusters.

          Show
          Suresh Srinivas added a comment - > ... "hadoop namenode -format mycluster". If you don't specify this it should print usage info. This is what is done now. > one may provide a cluster ID with "hadoop namenode -upgrade -clusterid foo This is what is done now. > if cluster ID is meant to be a user-visible "nice name" – how can one rename a cluster? Currently there is not functionality to do this. But we may need to add this to facilitate merging of clusters.
          Hide
          Sanjay Radia added a comment -

          I think we are mixing the nice name with the original motivation of clusterid: clusterId ensures that the system automatically
          detects misconfigured DNs and NN by the clusterid (as the namespaceid used to do).
          We recently had an operator who did that; the system detected this and told him that the namespace ids do not match. The operator then did a sshall to all datanodes to change the namespace id; of course the NN accepted the "wrong DNs" and promptly started to delete the blocks.
          A more descriptive name would have helped. But lets not mix the two.

          With Todd's q on rennaming the nice name I realize that a smart operator may decided that he wants to switch the names of 2 clusters and therefore does a quick rename on both NNs. If he makes a mistake about which DNs are connected to which cluster a disaster could occur. A clusterId is a birthmark – keep it fixed forever unless you want to merge two clusters. Keep the nice name separate from the clusterid.

          Show
          Sanjay Radia added a comment - I think we are mixing the nice name with the original motivation of clusterid: clusterId ensures that the system automatically detects misconfigured DNs and NN by the clusterid (as the namespaceid used to do). We recently had an operator who did that; the system detected this and told him that the namespace ids do not match. The operator then did a sshall to all datanodes to change the namespace id; of course the NN accepted the "wrong DNs" and promptly started to delete the blocks. A more descriptive name would have helped. But lets not mix the two. With Todd's q on rennaming the nice name I realize that a smart operator may decided that he wants to switch the names of 2 clusters and therefore does a quick rename on both NNs. If he makes a mistake about which DNs are connected to which cluster a disaster could occur. A clusterId is a birthmark – keep it fixed forever unless you want to merge two clusters. Keep the nice name separate from the clusterid.
          Hide
          Suresh Srinivas added a comment -

          Sanjay, I think our data files that are in plain text, should not be so, to avoid people messing with it.

          We should add in this jira, to the description of the command, the uniqueness requirement of cluster ID.

          Show
          Suresh Srinivas added a comment - Sanjay, I think our data files that are in plain text, should not be so, to avoid people messing with it. We should add in this jira, to the description of the command, the uniqueness requirement of cluster ID.
          Hide
          Sanjay Radia added a comment -

          Fixing the data files so that they are not plain text will prevent an operator from updating the "superblock". I was using
          the above example to illustrate there is a need for a nice cluster name since the operator can then see that he is trying to
          connect a DN of ClusterFoo to NN in ClusterBar. He realizes the error and fixes it rather then trying to figure what cluster id belongs to which cluster.

          My 2nd point is that giving a nice name to the clusterid prompts the question of renaming it – this runs into further problems.
          My argument is that keep the two separate. ClusterId is a birthmark that has be globaly unique - it is fixed unless one is merging clusters. I was willing to live with letting the oeprator supply a name that he gaurantees to be globally unique within his data centers; unfortunately a human readabale cluster id begs a command to rename the clusterid.

          Folks are too concerned about an extra parameter at the time of formatting a NN – this is an infrequent operation. Don't muck around with the design and architecture of the system for this very infrequent operation. I have no problem with the format command generating a cluster id when one is not given. This takes care of the users who only want one NN.

          I am not sure where to store the nice cluster name. Perhaps a mapping file?

          Show
          Sanjay Radia added a comment - Fixing the data files so that they are not plain text will prevent an operator from updating the "superblock". I was using the above example to illustrate there is a need for a nice cluster name since the operator can then see that he is trying to connect a DN of ClusterFoo to NN in ClusterBar. He realizes the error and fixes it rather then trying to figure what cluster id belongs to which cluster. My 2nd point is that giving a nice name to the clusterid prompts the question of renaming it – this runs into further problems. My argument is that keep the two separate. ClusterId is a birthmark that has be globaly unique - it is fixed unless one is merging clusters. I was willing to live with letting the oeprator supply a name that he gaurantees to be globally unique within his data centers; unfortunately a human readabale cluster id begs a command to rename the clusterid. Folks are too concerned about an extra parameter at the time of formatting a NN – this is an infrequent operation. Don't muck around with the design and architecture of the system for this very infrequent operation. I have no problem with the format command generating a cluster id when one is not given. This takes care of the users who only want one NN. I am not sure where to store the nice cluster name. Perhaps a mapping file?
          Hide
          Doug Balog added a comment -

          > Folks are too concerned about an extra parameter at the time of formatting a NN – this is an infrequent operation.

          I think you are overlooking the fact that there is a lot of documentation/tutorials on setting up a hadoop
          cluster that you just broke for a feature that most people probably aren't going to use.
          Make the people who want to use Federation add an extra parameter for each additional NN
          they want to add to their cluster.

          Show
          Doug Balog added a comment - > Folks are too concerned about an extra parameter at the time of formatting a NN – this is an infrequent operation. I think you are overlooking the fact that there is a lot of documentation/tutorials on setting up a hadoop cluster that you just broke for a feature that most people probably aren't going to use. Make the people who want to use Federation add an extra parameter for each additional NN they want to add to their cluster.
          Hide
          Suresh Srinivas added a comment -

          Doug, please read the previous comments on the rationale.

          This is one command changing. It will be documented. While this is a change, I am not sure this is a huge complexity.

          Show
          Suresh Srinivas added a comment - Doug, please read the previous comments on the rationale. This is one command changing. It will be documented. While this is a change, I am not sure this is a huge complexity.
          Hide
          Sanjay Radia added a comment -

          The format command prints a usage message explaining the usage so the issue is not as big as it is made out.
          The notion of a cluster id will become more and more important over time and goes beyond federation; it existed in the hadoop from early days but the design was not completely correct. But lets not debate that; instead we will change the NN format command to generate one if it is not supplied. This will maintain the compatibility for this operator CLI.
          Are folks happy with that?

          Show
          Sanjay Radia added a comment - The format command prints a usage message explaining the usage so the issue is not as big as it is made out. The notion of a cluster id will become more and more important over time and goes beyond federation; it existed in the hadoop from early days but the design was not completely correct. But lets not debate that; instead we will change the NN format command to generate one if it is not supplied. This will maintain the compatibility for this operator CLI. Are folks happy with that?
          Hide
          Todd Lipcon added a comment -

          Sounds good to me, Sanjay.

          Show
          Todd Lipcon added a comment - Sounds good to me, Sanjay.
          Hide
          Eli Collins added a comment -

          Sounds good to me as well.

          Show
          Eli Collins added a comment - Sounds good to me as well.
          Hide
          Bharath Mundlapudi added a comment -

          Attached the patch which addressing the following:

          1. clusterid will be automatically generated if user doesn't provide one.
          2. Admins can specify clusterid with -clusterid option.

          Show
          Bharath Mundlapudi added a comment - Attached the patch which addressing the following: 1. clusterid will be automatically generated if user doesn't provide one. 2. Admins can specify clusterid with -clusterid option.
          Hide
          Suresh Srinivas added a comment -

          Comments:

          1. Namenode#format() returns true if formatting was aborted. Not sure why you changed the return value from the method. It now does the opposite of what the method javadoc says.
          2. Please remove reusing the clusterID from an old unrelated image during format. This is unnecessary.
          3. TestClusterId test will fail with this change. Need changes in that test.

          Equivalent change is needed in upgrade command to handle the cluster ID similarly.

          Show
          Suresh Srinivas added a comment - Comments: Namenode#format() returns true if formatting was aborted. Not sure why you changed the return value from the method. It now does the opposite of what the method javadoc says. Please remove reusing the clusterID from an old unrelated image during format. This is unnecessary. TestClusterId test will fail with this change. Need changes in that test. Equivalent change is needed in upgrade command to handle the cluster ID similarly.
          Hide
          Bharath Mundlapudi added a comment -

          Thanks for the review, Suresh.

          My comments:
          1 & 3. From a high level, format returns boolean. Semantically, if this operation was successful, we should return true else we should return false.

          The previous code was bit not right. If format was successful, it was returning false. Even if user opts to not format, format operation as such was not successful, so we should return false. So, i changed this part also.

          Let me know, if these assumptions are not correct, i will back to previous semantics.

          I will fix the doc and tests. Sorry, i missed this part.

          Regarding upgrade, do you want me to do in another Jira? Since, this was just filed for format usability.

          Show
          Bharath Mundlapudi added a comment - Thanks for the review, Suresh. My comments: 1 & 3. From a high level, format returns boolean. Semantically, if this operation was successful, we should return true else we should return false. The previous code was bit not right. If format was successful, it was returning false. Even if user opts to not format, format operation as such was not successful, so we should return false. So, i changed this part also. Let me know, if these assumptions are not correct, i will back to previous semantics. I will fix the doc and tests. Sorry, i missed this part. Regarding upgrade, do you want me to do in another Jira? Since, this was just filed for format usability.
          Hide
          Bharath Mundlapudi added a comment -

          For Comment 2: Lets say that, i want to format a namenode which is part of a particular cluster. Reusing the cluster id is useful here - I just want to format this namenode and also want it to be part of same cluster.

          Show
          Bharath Mundlapudi added a comment - For Comment 2: Lets say that, i want to format a namenode which is part of a particular cluster. Reusing the cluster id is useful here - I just want to format this namenode and also want it to be part of same cluster.
          Hide
          Suresh Srinivas added a comment -

          > Semantically, if this operation was successful, we should return true else we should return false.
          Currently format() method returns aborted state instead of completion state. Given that definition, format() method seems fine.

          As regards to reusing clusterId - I feel format wipes out the previous state of the namenode. Reusing cluster ID adds unnecessary Y/N questions and makes the behaviors complicated to document.

          Show
          Suresh Srinivas added a comment - > Semantically, if this operation was successful, we should return true else we should return false. Currently format() method returns aborted state instead of completion state. Given that definition, format() method seems fine. As regards to reusing clusterId - I feel format wipes out the previous state of the namenode. Reusing cluster ID adds unnecessary Y/N questions and makes the behaviors complicated to document.
          Hide
          Bharath Mundlapudi added a comment -

          Attaching a patch based on comments. Preserved the previous semantics.

          Show
          Bharath Mundlapudi added a comment - Attaching a patch based on comments. Preserved the previous semantics.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479685/HDFS-1905-2.patch
          against trunk revision 1124459.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
          org.apache.hadoop.hdfs.TestFileConcurrentReader
          org.apache.hadoop.tools.TestJMXGet

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/583//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/583//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/583//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479685/HDFS-1905-2.patch against trunk revision 1124459. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.tools.TestJMXGet +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/583//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/583//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/583//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479685/HDFS-1905-2.patch
          against trunk revision 1124459.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
          org.apache.hadoop.hdfs.TestFileConcurrentReader
          org.apache.hadoop.hdfs.TestGetBlocks
          org.apache.hadoop.hdfs.TestHDFSTrash
          org.apache.hadoop.tools.TestJMXGet

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/584//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/584//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/584//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479685/HDFS-1905-2.patch against trunk revision 1124459. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.hdfs.TestGetBlocks org.apache.hadoop.hdfs.TestHDFSTrash org.apache.hadoop.tools.TestJMXGet +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/584//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/584//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/584//console This message is automatically generated.
          Hide
          Suresh Srinivas added a comment -

          +1 for the patch.

          Show
          Suresh Srinivas added a comment - +1 for the patch.
          Hide
          Suresh Srinivas added a comment -

          I committed the patch. Thank you Bharath.

          Show
          Suresh Srinivas added a comment - I committed the patch. Thank you Bharath.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #677 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/677/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #677 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/677/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #673 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/673/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #673 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/673/ )
          Hide
          Suresh Srinivas added a comment -

          While looking at our automated deployments - which are failing due to this change, I thought of an issue with this change.

          In this change, for non federation users we decide to "namenode -format" command should automatically generated cluster ID. The problem is for the secondary/backup/checkpointer, one must run "namenode -format -clusterid <cid>", where cid is from the first namenode that is formatted. Without this, the second namenode would generate its own cluster ID and will not be part of the same cluster as that of primary.

          I vote for reverting this change and move to "namenode -format -cluster <cid>".

          Show
          Suresh Srinivas added a comment - While looking at our automated deployments - which are failing due to this change, I thought of an issue with this change. In this change, for non federation users we decide to "namenode -format" command should automatically generated cluster ID. The problem is for the secondary/backup/checkpointer, one must run "namenode -format -clusterid <cid>", where cid is from the first namenode that is formatted. Without this, the second namenode would generate its own cluster ID and will not be part of the same cluster as that of primary. I vote for reverting this change and move to "namenode -format -cluster <cid>".
          Hide
          Todd Lipcon added a comment -

          IMO that is an issue with the secondary/backup/checkpointer – you shouldn't have to format it at all. When it's started with an empty namespace, it should simply grab the clusterID/etc from the NN that it's configured to checkpoint from. This is how it has worked in the past with namespace ID, for example - why should cluster ID be treated differently?

          Show
          Todd Lipcon added a comment - IMO that is an issue with the secondary/backup/checkpointer – you shouldn't have to format it at all. When it's started with an empty namespace, it should simply grab the clusterID/etc from the NN that it's configured to checkpoint from. This is how it has worked in the past with namespace ID, for example - why should cluster ID be treated differently?
          Hide
          Suresh Srinivas added a comment -

          Some one here corrected me that format is not needed for these nodes. So we do not have to revert this patch.

          Show
          Suresh Srinivas added a comment - Some one here corrected me that format is not needed for these nodes. So we do not have to revert this patch.

            People

            • Assignee:
              Bharath Mundlapudi
              Reporter:
              Bharath Mundlapudi
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development