Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.3-alpha
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      DataStorage.java

      • The "initilized" member is misspelled
      • Comment what the storageID member is

      DataNode.java

      • Cleanup createNewStorageId comment (should mention the port is included and is overly verbose)

      BlockManager.java

      • TreeSet in the comment should be TreeMap
      1. hdfs-4377.txt
        6 kB
        Eli Collins
      2. hdfs-4377.txt
        5 kB
        Eli Collins

        Activity

        Hide
        Eli Collins added a comment -

        Patch attached.

        Show
        Eli Collins added a comment - Patch attached.
        Hide
        Todd Lipcon added a comment -
        -  // Keeps a TreeSet for every named node. Each treeset contains
        +  // Keeps a TreeMap for every named node. Each treeset contains
           // a list of the blocks that are "extra" at that location. We'll
           // eventually remove these extras.
        -  // Mapping: StorageID -> TreeSet<Block>
        +  // Mapping: StorageID -> TreeMap<Block>
           //
           public final Map<String, LightWeightLinkedSet<Block>> excessReplicateMap =
             new TreeMap<String, LightWeightLinkedSet<Block>>();
        

        This isn't right – the value of the map is itself a set, not a map.


        -     * It is considered extermely rare for all these numbers to match
        -     * on a different machine accidentally for the following 
        -     * a) SecureRandom(INT_MAX) is pretty much random (1 in 2 billion), and
        -     * b) Good chance ip address would be different, and
        -     * c) Even on the same machine, Datanode is designed to use different ports.
        -     * d) Good chance that these are started at different times.
        -     * For a confict to occur all the 4 above have to match!.
        -     * The format of this string can be changed anytime in future without
        -     * affecting its functionality.
        

        I think those comments were useful. Probably worth leaving them as '//' style comments.


        Otherwise seems fine

        Show
        Todd Lipcon added a comment - - // Keeps a TreeSet for every named node. Each treeset contains + // Keeps a TreeMap for every named node. Each treeset contains // a list of the blocks that are "extra" at that location. We'll // eventually remove these extras. - // Mapping: StorageID -> TreeSet<Block> + // Mapping: StorageID -> TreeMap<Block> // public final Map< String , LightWeightLinkedSet<Block>> excessReplicateMap = new TreeMap< String , LightWeightLinkedSet<Block>>(); This isn't right – the value of the map is itself a set, not a map. - * It is considered extermely rare for all these numbers to match - * on a different machine accidentally for the following - * a) SecureRandom(INT_MAX) is pretty much random (1 in 2 billion), and - * b) Good chance ip address would be different, and - * c) Even on the same machine, Datanode is designed to use different ports. - * d) Good chance that these are started at different times. - * For a confict to occur all the 4 above have to match!. - * The format of this string can be changed anytime in future without - * affecting its functionality. I think those comments were useful. Probably worth leaving them as '//' style comments. Otherwise seems fine
        Hide
        Eli Collins added a comment -

        Thanks for the review Todd. Updated patch attached.

        #1 Good catch, I just noticed the type changed recently and updated the comment naively. I re-wrote it now. Related, I filed HDFS-4378 to make StorageID a class so the types are more readable and the code is less error prone.

        #2 I rewrote the implementation comment to fix spelling mistakes/grammar and hopefully improve the explanation.

        Show
        Eli Collins added a comment - Thanks for the review Todd. Updated patch attached. #1 Good catch, I just noticed the type changed recently and updated the comment naively. I re-wrote it now. Related, I filed HDFS-4378 to make StorageID a class so the types are more readable and the code is less error prone. #2 I rewrote the implementation comment to fix spelling mistakes/grammar and hopefully improve the explanation.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12563989/hdfs-4377.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3808//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3808//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563989/hdfs-4377.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3808//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3808//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12564025/hdfs-4377.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3809//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3809//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12564025/hdfs-4377.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3809//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3809//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12564025/hdfs-4377.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.TestPersistBlocks

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3810//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3810//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12564025/hdfs-4377.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestPersistBlocks +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3810//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3810//console This message is automatically generated.
        Hide
        Eli Collins added a comment -

        Test failures are unrelated. Good to go Todd?

        Show
        Eli Collins added a comment - Test failures are unrelated. Good to go Todd?
        Hide
        Todd Lipcon added a comment -

        +1

        Show
        Todd Lipcon added a comment - +1
        Hide
        Eli Collins added a comment -

        Thanks Todd, I've committed this and merged to branch-2.

        Show
        Eli Collins added a comment - Thanks Todd, I've committed this and merged to branch-2.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #3215 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3215/)
        HDFS-4377. Some trivial DN comment cleanup. Contributed by Eli Collins (Revision 1431753)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431753
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #3215 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3215/ ) HDFS-4377 . Some trivial DN comment cleanup. Contributed by Eli Collins (Revision 1431753) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431753 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #93 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/93/)
        HDFS-4377. Some trivial DN comment cleanup. Contributed by Eli Collins (Revision 1431753)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431753
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #93 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/93/ ) HDFS-4377 . Some trivial DN comment cleanup. Contributed by Eli Collins (Revision 1431753) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431753 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1282 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1282/)
        HDFS-4377. Some trivial DN comment cleanup. Contributed by Eli Collins (Revision 1431753)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431753
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1282 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1282/ ) HDFS-4377 . Some trivial DN comment cleanup. Contributed by Eli Collins (Revision 1431753) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431753 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1310 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1310/)
        HDFS-4377. Some trivial DN comment cleanup. Contributed by Eli Collins (Revision 1431753)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431753
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1310 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1310/ ) HDFS-4377 . Some trivial DN comment cleanup. Contributed by Eli Collins (Revision 1431753) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431753 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java

          People

          • Assignee:
            Eli Collins
            Reporter:
            Eli Collins
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development