Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Per Colin P. McCabe's comments in HDFS-8792, this JIRA is to discuss improving BlockManager#excessReplicateMap.

      That's right HashMap don't ever shrink when elements are removed, but TreeMap entry needs to store more (memory) references (left, right, parent) than HashMap entry (only one reference next), even when there is element removing and cause some entry empty, the empty HashMap entry is just a null reference (4 bytes), so they are close at this point. On the other hand, the key of excessReplicateMap is datanode uuid, so the entries number is almost fixed, so HashMap memory is good than TreeMap memory in this case. I think the most important is the search/insert/remove performance, HashMap is absolutely better than TreeMap. Because we don't need to sort, we should use HashMap instead of TreeMap

        Activity

        Hide
        cmccabe Colin P. McCabe added a comment -

        Thanks, Yi Liu.

        Show
        cmccabe Colin P. McCabe added a comment - Thanks, Yi Liu .
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #280 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/280/)
        HDFS-8862. BlockManager#excessReplicateMap should use a HashMap. (yliu) (yliu: rev 71566e23820d33e0110ca55eded3299735e970b9)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #280 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/280/ ) HDFS-8862 . BlockManager#excessReplicateMap should use a HashMap. (yliu) (yliu: rev 71566e23820d33e0110ca55eded3299735e970b9) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2218 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2218/)
        HDFS-8862. BlockManager#excessReplicateMap should use a HashMap. (yliu) (yliu: rev 71566e23820d33e0110ca55eded3299735e970b9)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2218 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2218/ ) HDFS-8862 . BlockManager#excessReplicateMap should use a HashMap. (yliu) (yliu: rev 71566e23820d33e0110ca55eded3299735e970b9) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2237 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2237/)
        HDFS-8862. BlockManager#excessReplicateMap should use a HashMap. (yliu) (yliu: rev 71566e23820d33e0110ca55eded3299735e970b9)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2237 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2237/ ) HDFS-8862 . BlockManager#excessReplicateMap should use a HashMap. (yliu) (yliu: rev 71566e23820d33e0110ca55eded3299735e970b9) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #288 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/288/)
        HDFS-8862. BlockManager#excessReplicateMap should use a HashMap. (yliu) (yliu: rev 71566e23820d33e0110ca55eded3299735e970b9)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #288 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/288/ ) HDFS-8862 . BlockManager#excessReplicateMap should use a HashMap. (yliu) (yliu: rev 71566e23820d33e0110ca55eded3299735e970b9) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #291 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/291/)
        HDFS-8862. BlockManager#excessReplicateMap should use a HashMap. (yliu) (yliu: rev 71566e23820d33e0110ca55eded3299735e970b9)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #291 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/291/ ) HDFS-8862 . BlockManager#excessReplicateMap should use a HashMap. (yliu) (yliu: rev 71566e23820d33e0110ca55eded3299735e970b9) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1021 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1021/)
        HDFS-8862. BlockManager#excessReplicateMap should use a HashMap. (yliu) (yliu: rev 71566e23820d33e0110ca55eded3299735e970b9)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1021 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1021/ ) HDFS-8862 . BlockManager#excessReplicateMap should use a HashMap. (yliu) (yliu: rev 71566e23820d33e0110ca55eded3299735e970b9) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8315 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8315/)
        HDFS-8862. BlockManager#excessReplicateMap should use a HashMap. (yliu) (yliu: rev 71566e23820d33e0110ca55eded3299735e970b9)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8315 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8315/ ) HDFS-8862 . BlockManager#excessReplicateMap should use a HashMap. (yliu) (yliu: rev 71566e23820d33e0110ca55eded3299735e970b9) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hitliuyi Yi Liu added a comment -

        Thanks Colin P. McCabe for the review. Will commit the patch later.

        LightWeightLinkedSet<BlockInfo> can shrink..

        Show
        hitliuyi Yi Liu added a comment - Thanks Colin P. McCabe for the review. Will commit the patch later. LightWeightLinkedSet<BlockInfo> can shrink..
        Hide
        cmccabe Colin P. McCabe added a comment -

        I guess the set of datanodes is not going to shrink that much over the life of the cluster. So the fact that this data structure can't shrink should be OK. We may want to look into whether that LightWeightLinkedSet<BlockInfo> can shrink... but that is outside the scope of this JIRA. +1.

        Show
        cmccabe Colin P. McCabe added a comment - I guess the set of datanodes is not going to shrink that much over the life of the cluster. So the fact that this data structure can't shrink should be OK. We may want to look into whether that LightWeightLinkedSet<BlockInfo> can shrink... but that is outside the scope of this JIRA. +1.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 15m 19s Findbugs (version ) appears to be broken on trunk.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 36s There were no new javac warning messages.
        +1 javadoc 9m 46s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 4s The applied patch generated 171 new checkstyle issues (total was 0, now 171).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 25s mvn install still works.
        +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
        +1 findbugs 2m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 3s Pre-build of native portion
        +1 hdfs tests 159m 37s Tests passed in hadoop-hdfs.
            201m 15s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12748980/HDFS-8862.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / f59612e
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11915/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11915/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11915/testReport/
        Java 1.7.0_55
        uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11915/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 15m 19s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 36s There were no new javac warning messages. +1 javadoc 9m 46s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 4s The applied patch generated 171 new checkstyle issues (total was 0, now 171). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 25s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 findbugs 2m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 3s Pre-build of native portion +1 hdfs tests 159m 37s Tests passed in hadoop-hdfs.     201m 15s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12748980/HDFS-8862.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f59612e checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11915/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11915/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11915/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11915/console This message was automatically generated.

          People

          • Assignee:
            hitliuyi Yi Liu
            Reporter:
            hitliuyi Yi Liu
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development