Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8859

Improve DataNode ReplicaMap memory footprint to save about 45%

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      By using following approach we can save about 45% memory footprint for each block replica in DataNode memory (This JIRA only talks about ReplicaMap in DataNode), the details are:

      In ReplicaMap,

      private final Map<String, Map<Long, ReplicaInfo>> map =
          new HashMap<String, Map<Long, ReplicaInfo>>();
      

      Currently we use a HashMap Map<Long, ReplicaInfo> to store the replicas in memory. The key is block id of the block replica which is already included in ReplicaInfo, so this memory can be saved. Also HashMap Entry has a object overhead. We can implement a lightweight Set which is similar to LightWeightGSet, but not a fixed size (LightWeightGSet uses fix size for the entries array, usually it's a big value, an example is BlocksMap, this can avoid full gc since no need to resize), also we should be able to get Element through key.

      Following is comparison of memory footprint If we implement a lightweight set as described:

      We can save:

          SIZE (bytes)           ITEM
          20                        The Key: Long (12 bytes object overhead + 8 bytes long)
          12                        HashMap Entry object overhead
          4                          reference to the key in Entry
          4                          reference to the value in Entry
          4                          hash in Entry
      

      Total: -44 bytes

      We need to add:

          SIZE (bytes)           ITEM
          4                             a reference to next element in ReplicaInfo
      

      Total: +4 bytes

      So totally we can save 40bytes for each block replica

      And currently one finalized replica needs around 46 bytes (notice: we ignore memory alignment here).

      We can save 1 - (4 + 46) / (44 + 46) = 45% memory for each block replica in DataNode.

      1. HDFS-8859.001.patch
        26 kB
        Yi Liu
      2. HDFS-8859.002.patch
        27 kB
        Yi Liu
      3. HDFS-8859.003.patch
        30 kB
        Yi Liu
      4. HDFS-8859.004.patch
        34 kB
        Yi Liu
      5. HDFS-8859.005.patch
        34 kB
        Yi Liu
      6. HDFS-8859.006.patch
        34 kB
        Yi Liu

        Activity

        Hide
        hitliuyi Yi Liu added a comment - - edited

        LightWeightHashGSet implemented in patch is a low memory footprint GSet implementation, which uses an array for storing the elements and linked lists for collision resolution. If the size of elements exceeds the threshold, the internal array will be resized to double length. Default load factor is 0.75f which is the same as java HashMap.

        Currently LightWeightHashGSet doesn't shrink when removing elements and hitting some threshold, I feel it's not necessary for our case. If you do think we'd better to have this, I can do it in a follow-on.

        As shown in the patch, ReplicaInfo needs to implement LightWeightHashGSet.LinkedElement now, and modification in ReplicaMap is to use this new lightweight set.

        By using the new light weight set, we can get the benefits (reduce a lot of DataNode (ReplicaMap) memory footprint) as described in the JIRA description.

        Please review, thanks.

        Show
        hitliuyi Yi Liu added a comment - - edited LightWeightHashGSet implemented in patch is a low memory footprint GSet implementation, which uses an array for storing the elements and linked lists for collision resolution. If the size of elements exceeds the threshold, the internal array will be resized to double length. Default load factor is 0.75f which is the same as java HashMap . Currently LightWeightHashGSet doesn't shrink when removing elements and hitting some threshold, I feel it's not necessary for our case. If you do think we'd better to have this, I can do it in a follow-on. As shown in the patch, ReplicaInfo needs to implement LightWeightHashGSet.LinkedElement now, and modification in ReplicaMap is to use this new lightweight set. By using the new light weight set, we can get the benefits (reduce a lot of DataNode (ReplicaMap) memory footprint) as described in the JIRA description. Please review, thanks.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 15m 46s Findbugs (version ) appears to be broken on trunk.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 41s There were no new javac warning messages.
        +1 javadoc 9m 40s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 14s There were no new checkstyle issues.
        -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 30s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 4m 24s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        -1 common tests 22m 31s Tests failed in hadoop-common.
        -1 hdfs tests 188m 11s Tests failed in hadoop-hdfs.
            251m 55s  



        Reason Tests
        Failed unit tests hadoop.net.TestNetUtils
          hadoop.ha.TestZKFailoverController
          hadoop.hdfs.server.namenode.TestParallelImageWrite
          hadoop.hdfs.TestFileAppend2
          hadoop.hdfs.server.namenode.snapshot.TestSnapshot
          hadoop.hdfs.server.namenode.ha.TestStandbyIsHot
          hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
          hadoop.hdfs.TestDFSUpgradeFromImage
          hadoop.hdfs.TestDatanodeLayoutUpgrade
          hadoop.hdfs.server.namenode.ha.TestDNFencing
        Timed out tests org.apache.hadoop.cli.TestHDFSCLI



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12749223/HDFS-8859.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / b6265d3
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11934/artifact/patchprocess/whitespace.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/11934/artifact/patchprocess/testrun_hadoop-common.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11934/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11934/testReport/
        Java 1.7.0_55
        uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11934/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 15m 46s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 41s There were no new javac warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 14s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 30s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 4m 24s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 22m 31s Tests failed in hadoop-common. -1 hdfs tests 188m 11s Tests failed in hadoop-hdfs.     251m 55s   Reason Tests Failed unit tests hadoop.net.TestNetUtils   hadoop.ha.TestZKFailoverController   hadoop.hdfs.server.namenode.TestParallelImageWrite   hadoop.hdfs.TestFileAppend2   hadoop.hdfs.server.namenode.snapshot.TestSnapshot   hadoop.hdfs.server.namenode.ha.TestStandbyIsHot   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.TestDatanodeLayoutUpgrade   hadoop.hdfs.server.namenode.ha.TestDNFencing Timed out tests org.apache.hadoop.cli.TestHDFSCLI Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12749223/HDFS-8859.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / b6265d3 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11934/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/11934/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11934/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11934/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11934/console This message was automatically generated.
        Hide
        hitliuyi Yi Liu added a comment -

        Fix the test failures and enhance the test.

        Show
        hitliuyi Yi Liu added a comment - Fix the test failures and enhance the test.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 19m 2s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 41s There were no new javac warning messages.
        +1 javadoc 9m 38s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 46s The applied patch generated 12 new checkstyle issues (total was 0, now 12).
        -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 29s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 4m 21s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        -1 common tests 22m 17s Tests failed in hadoop-common.
        -1 hdfs tests 175m 43s Tests failed in hadoop-hdfs.
            243m 11s  



        Reason Tests
        Failed unit tests hadoop.ha.TestZKFailoverController
          hadoop.net.TestNetUtils
        Timed out tests org.apache.hadoop.cli.TestHDFSCLI



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12749402/HDFS-8859.002.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 8f73bdd
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11947/artifact/patchprocess/diffcheckstylehadoop-common.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11947/artifact/patchprocess/whitespace.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/11947/artifact/patchprocess/testrun_hadoop-common.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11947/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11947/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11947/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 2s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 41s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 46s The applied patch generated 12 new checkstyle issues (total was 0, now 12). -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 21s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 22m 17s Tests failed in hadoop-common. -1 hdfs tests 175m 43s Tests failed in hadoop-hdfs.     243m 11s   Reason Tests Failed unit tests hadoop.ha.TestZKFailoverController   hadoop.net.TestNetUtils Timed out tests org.apache.hadoop.cli.TestHDFSCLI Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12749402/HDFS-8859.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 8f73bdd checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11947/artifact/patchprocess/diffcheckstylehadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11947/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/11947/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11947/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11947/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11947/console This message was automatically generated.
        Hide
        hitliuyi Yi Liu added a comment -

        The two test failures are not related.

        Show
        hitliuyi Yi Liu added a comment - The two test failures are not related.
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -
        • Is the only difference between LightWeightHashGSet and LightWeightGSet that LightWeightHashGSet is resizable?
        • It seems that some code in LightWeightHashGSet is copied from LightWeightGSet. Could you change LightWeightHashGSet to extends LightWeightGSet?
        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - Is the only difference between LightWeightHashGSet and LightWeightGSet that LightWeightHashGSet is resizable? It seems that some code in LightWeightHashGSet is copied from LightWeightGSet. Could you change LightWeightHashGSet to extends LightWeightGSet?
        Hide
        hitliuyi Yi Liu added a comment -

        Thanks Tsz Wo Nicholas Sze for the review.
        For your first question, yes, and another small difference is in LightWeightHashGSet needs to implement public Collection<E> values() as java HashMap, now I add it as an interface of GSet

        For your second comment, you are right, it's more better to change LightWeightHashGSet extends LightWeightGSet, I do it in the new patch. Actually when I made the first patch, I ever considered make LightWeightHashGSet to extend LightWeightGSet, at that time I thought to support shrink later and more logic may be different, and make them independent. But I agree we should extend even so.

        Show
        hitliuyi Yi Liu added a comment - Thanks Tsz Wo Nicholas Sze for the review. For your first question, yes, and another small difference is in LightWeightHashGSet needs to implement public Collection<E> values() as java HashMap, now I add it as an interface of GSet For your second comment, you are right, it's more better to change LightWeightHashGSet extends LightWeightGSet, I do it in the new patch. Actually when I made the first patch, I ever considered make LightWeightHashGSet to extend LightWeightGSet, at that time I thought to support shrink later and more logic may be different, and make them independent. But I agree we should extend even so.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 17m 2s Findbugs (version ) appears to be broken on trunk.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
        +1 javac 7m 44s There were no new javac warning messages.
        +1 javadoc 9m 38s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 29s The applied patch generated 6 new checkstyle issues (total was 12, now 14).
        -1 whitespace 0m 1s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 29s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 4m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        -1 common tests 22m 16s Tests failed in hadoop-common.
        -1 hdfs tests 220m 50s Tests failed in hadoop-hdfs.
            286m 7s  



        Reason Tests
        Failed unit tests hadoop.ha.TestZKFailoverController
          hadoop.net.TestNetUtils
          hadoop.hdfs.server.namenode.ha.TestDNFencing
          hadoop.hdfs.server.namenode.snapshot.TestSnapshot
        Timed out tests org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite
          org.apache.hadoop.hdfs.server.namenode.TestFsck
          org.apache.hadoop.hdfs.TestRestartDFS
          org.apache.hadoop.cli.TestHDFSCLI



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12750016/HDFS-8859.003.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 1ea1a83
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11976/artifact/patchprocess/diffcheckstylehadoop-common.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11976/artifact/patchprocess/whitespace.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/11976/artifact/patchprocess/testrun_hadoop-common.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11976/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11976/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11976/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 2s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 44s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 29s The applied patch generated 6 new checkstyle issues (total was 12, now 14). -1 whitespace 0m 1s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 4m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 22m 16s Tests failed in hadoop-common. -1 hdfs tests 220m 50s Tests failed in hadoop-hdfs.     286m 7s   Reason Tests Failed unit tests hadoop.ha.TestZKFailoverController   hadoop.net.TestNetUtils   hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.server.namenode.snapshot.TestSnapshot Timed out tests org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite   org.apache.hadoop.hdfs.server.namenode.TestFsck   org.apache.hadoop.hdfs.TestRestartDFS   org.apache.hadoop.cli.TestHDFSCLI Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12750016/HDFS-8859.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 1ea1a83 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11976/artifact/patchprocess/diffcheckstylehadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11976/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/11976/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11976/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11976/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11976/console This message was automatically generated.
        Hide
        hitliuyi Yi Liu added a comment -

        TestRestartDFS is related to 003 patch, but 002 doesn't cause any issue. I just debug it, the reason seems the original implementation of SetIterator in LightWeightGSet has some issue, I wrote a more clear SetIterator in the new class LightWeightHashGSet in 002, but in 003, I make it to extend LightWeightGSet but not use my new implementation of SetIterator. If I use my new implementation of SetIterator, then the failure disappears.

        Let me find some time later to see why original implementation of SetIterator in LightWeightGSet causes the failure (it was not used in original code, so the bug might not be found).

        Show
        hitliuyi Yi Liu added a comment - TestRestartDFS is related to 003 patch, but 002 doesn't cause any issue. I just debug it, the reason seems the original implementation of SetIterator in LightWeightGSet has some issue, I wrote a more clear SetIterator in the new class LightWeightHashGSet in 002 , but in 003 , I make it to extend LightWeightGSet but not use my new implementation of SetIterator . If I use my new implementation of SetIterator , then the failure disappears. Let me find some time later to see why original implementation of SetIterator in LightWeightGSet causes the failure (it was not used in original code, so the bug might not be found).
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        The idea sound good. Some comments:

        • Both LightWeightGSet and the new LightWeightHashGSet use hash functions. So LightWeightHashGSet seems not a good name. How about calling it LightWeightResizableGSet?
        • From your calculation, the patch improve each block replica object size about 45%. The JIRA summary is misleading. It seems claiming that it improves the overall DataNode memory footprint by about 45%. For 10m replicas, the original overall map entry object size is ~900 MB and the new size is ~500MB. Is it correct?
        • Why adding LightWeightGSet.putElement? Subclass can call super.put(..).
        • There is a rewrite for LightWeightGSet.remove(..). Why? The old code is well tested. Please do not change it if possible.
        • Took a quick looks at the tests. I think we need some long running tests to make sure the correctness. See TestGSet.runMultipleTestGSet().
        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - The idea sound good. Some comments: Both LightWeightGSet and the new LightWeightHashGSet use hash functions. So LightWeightHashGSet seems not a good name. How about calling it LightWeightResizableGSet? From your calculation, the patch improve each block replica object size about 45%. The JIRA summary is misleading. It seems claiming that it improves the overall DataNode memory footprint by about 45%. For 10m replicas, the original overall map entry object size is ~900 MB and the new size is ~500MB. Is it correct? Why adding LightWeightGSet.putElement? Subclass can call super.put(..). There is a rewrite for LightWeightGSet.remove(..). Why? The old code is well tested. Please do not change it if possible. Took a quick looks at the tests. I think we need some long running tests to make sure the correctness. See TestGSet.runMultipleTestGSet().
        Hide
        hitliuyi Yi Liu added a comment - - edited

        Thanks Tsz Wo Nicholas Sze for the review! Update the patch to address your comments.

        How about calling it LightWeightResizableGSet?

        Agree, rename it in the new patch.

        From your calculation, the patch improve each block replica object size about 45%. The JIRA summary is misleading. It seems claiming that it improves the overall DataNode memory footprint by about 45%. For 10m replicas, the original overall map entry object size is ~900 MB and the new size is ~500MB. Is it correct?

        It's correct. I did add ReplicaMap in the JIRA summary, yes, I use (), , considering the ReplicaMap is the major long-lived object in memory of Datanode which could be large, of course, there are other aspects (many are transient: data read/write buffer, rpc buffer, etc..), I just highlighted the improvement.
        Let me remove the ().

        Subclass can call super.put(..)

        Update in the new patch. I just used to a new internal method .

        There is a rewrite for LightWeightGSet.remove(..)

        I revert it in the new patch and keep original one. Original implement has duplicate logic, we can share same logic for all the if...else...

        I think we need some long running tests to make sure the correctness. See TestGSet.runMultipleTestGSet()

        Agree, updated it in the new patch.

        For the test failures of 003, it's because there is one place (BlockPoolSlice) add replicaInfo to replicaMap from a tmp replicapMap, but the replicaInfo is still in the tmp one, we can remove it from the tmp one before adding (for LightWeightGSet, an element is not allowed to exist in two gset). In 002 patch, the failure didn't exist, we had a new implement of SetIterator which was very similar to the logic in java Hashmap, and a bit different with original one. But both are correct, the major difference is the time of finding next element. In the new patch, I keep the original one, and make few change in BlockPoolSlice. All tests run successfully in my local for the new patch.

        Show
        hitliuyi Yi Liu added a comment - - edited Thanks Tsz Wo Nicholas Sze for the review! Update the patch to address your comments. How about calling it LightWeightResizableGSet? Agree, rename it in the new patch. From your calculation, the patch improve each block replica object size about 45%. The JIRA summary is misleading. It seems claiming that it improves the overall DataNode memory footprint by about 45%. For 10m replicas, the original overall map entry object size is ~900 MB and the new size is ~500MB. Is it correct? It's correct. I did add ReplicaMap in the JIRA summary, yes, I use () , , considering the ReplicaMap is the major long-lived object in memory of Datanode which could be large, of course, there are other aspects (many are transient: data read/write buffer, rpc buffer, etc..), I just highlighted the improvement. Let me remove the () . Subclass can call super.put(..) Update in the new patch. I just used to a new internal method . There is a rewrite for LightWeightGSet.remove(..) I revert it in the new patch and keep original one. Original implement has duplicate logic, we can share same logic for all the if...else.. . I think we need some long running tests to make sure the correctness. See TestGSet.runMultipleTestGSet() Agree, updated it in the new patch. For the test failures of 003 , it's because there is one place (BlockPoolSlice) add replicaInfo to replicaMap from a tmp replicapMap, but the replicaInfo is still in the tmp one, we can remove it from the tmp one before adding (for LightWeightGSet, an element is not allowed to exist in two gset). In 002 patch, the failure didn't exist, we had a new implement of SetIterator which was very similar to the logic in java Hashmap, and a bit different with original one. But both are correct, the major difference is the time of finding next element. In the new patch, I keep the original one, and make few change in BlockPoolSlice. All tests run successfully in my local for the new patch.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 19m 21s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
        +1 javac 7m 52s There were no new javac warning messages.
        +1 javadoc 9m 51s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 50s The applied patch generated 6 new checkstyle issues (total was 12, now 16).
        -1 whitespace 0m 1s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 33s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 4m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        -1 common tests 22m 33s Tests failed in hadoop-common.
        -1 hdfs tests 76m 49s Tests failed in hadoop-hdfs.
            145m 35s  



        Reason Tests
        Failed unit tests hadoop.ha.TestZKFailoverController
          hadoop.net.TestNetUtils
          hadoop.hdfs.TestReplication
          hadoop.hdfs.TestSafeMode
          hadoop.hdfs.TestDatanodeRegistration
          hadoop.hdfs.tools.TestDebugAdmin
          hadoop.hdfs.TestSetrepIncreasing
          hadoop.hdfs.TestDatanodeReport
          hadoop.hdfs.TestDFSShellGenericOptions
          hadoop.hdfs.TestParallelRead
          hadoop.hdfs.tools.TestStoragePolicyCommands
          hadoop.hdfs.TestDFSRemove
          hadoop.hdfs.qjournal.TestSecureNNWithQJM
          hadoop.hdfs.web.TestWebHdfsTokens
          hadoop.hdfs.TestHFlush
          hadoop.hdfs.TestPersistBlocks
          hadoop.hdfs.TestParallelShortCircuitReadNoChecksum
          hadoop.hdfs.TestEncryptedTransfer
          hadoop.hdfs.TestQuota
          hadoop.hdfs.TestDFSClientFailover
          hadoop.hdfs.shortcircuit.TestShortCircuitCache
          hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl
          hadoop.hdfs.tools.TestDFSAdmin
          hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
          hadoop.hdfs.web.TestWebHdfsFileSystemContract
          hadoop.hdfs.web.TestWebHDFS
          hadoop.hdfs.TestFileAppend
          hadoop.hdfs.TestFileLengthOnClusterRestart
          hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary
          hadoop.hdfs.TestFSOutputSummer
          hadoop.hdfs.TestEncryptionZonesWithHA
          hadoop.hdfs.TestBlockReaderFactory
          hadoop.hdfs.TestDFSFinalize
          hadoop.hdfs.TestDisableConnCache
          hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes
          hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForXAttr
          hadoop.hdfs.web.TestHttpsFileSystem
          hadoop.hdfs.web.TestWebHdfsWithAuthenticationFilter
          hadoop.hdfs.web.TestWebHDFSAcl
          hadoop.hdfs.TestHDFSTrash
          hadoop.hdfs.TestDistributedFileSystem
          hadoop.hdfs.TestDataTransferKeepalive
          hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
          hadoop.hdfs.web.TestWebHDFSForHA
          hadoop.hdfs.TestBlockMissingException
          hadoop.hdfs.TestPipelines
          hadoop.hdfs.TestRenameWhileOpen
          hadoop.hdfs.TestFileCreationClient
          hadoop.hdfs.TestEncryptionZones
          hadoop.hdfs.TestFileAppend3
          hadoop.hdfs.TestBalancerBandwidth
          hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          hadoop.hdfs.TestSeekBug
          hadoop.hdfs.TestParallelShortCircuitReadUnCached
          hadoop.hdfs.TestBlockReaderLocal
          hadoop.hdfs.TestListFilesInFileContext
          hadoop.hdfs.web.TestWebHDFSXAttr
          hadoop.hdfs.TestFileStatus
          hadoop.hdfs.web.TestFSMainOperationsWebHdfs
        Timed out tests org.apache.hadoop.hdfs.TestFileCreation



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12750254/HDFS-8859.004.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 53bef9c
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11987/artifact/patchprocess/diffcheckstylehadoop-common.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11987/artifact/patchprocess/whitespace.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/11987/artifact/patchprocess/testrun_hadoop-common.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11987/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11987/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11987/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 21s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 52s There were no new javac warning messages. +1 javadoc 9m 51s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 50s The applied patch generated 6 new checkstyle issues (total was 12, now 16). -1 whitespace 0m 1s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 22m 33s Tests failed in hadoop-common. -1 hdfs tests 76m 49s Tests failed in hadoop-hdfs.     145m 35s   Reason Tests Failed unit tests hadoop.ha.TestZKFailoverController   hadoop.net.TestNetUtils   hadoop.hdfs.TestReplication   hadoop.hdfs.TestSafeMode   hadoop.hdfs.TestDatanodeRegistration   hadoop.hdfs.tools.TestDebugAdmin   hadoop.hdfs.TestSetrepIncreasing   hadoop.hdfs.TestDatanodeReport   hadoop.hdfs.TestDFSShellGenericOptions   hadoop.hdfs.TestParallelRead   hadoop.hdfs.tools.TestStoragePolicyCommands   hadoop.hdfs.TestDFSRemove   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.web.TestWebHdfsTokens   hadoop.hdfs.TestHFlush   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.TestParallelShortCircuitReadNoChecksum   hadoop.hdfs.TestEncryptedTransfer   hadoop.hdfs.TestQuota   hadoop.hdfs.TestDFSClientFailover   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl   hadoop.hdfs.tools.TestDFSAdmin   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.web.TestWebHdfsFileSystemContract   hadoop.hdfs.web.TestWebHDFS   hadoop.hdfs.TestFileAppend   hadoop.hdfs.TestFileLengthOnClusterRestart   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary   hadoop.hdfs.TestFSOutputSummer   hadoop.hdfs.TestEncryptionZonesWithHA   hadoop.hdfs.TestBlockReaderFactory   hadoop.hdfs.TestDFSFinalize   hadoop.hdfs.TestDisableConnCache   hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForXAttr   hadoop.hdfs.web.TestHttpsFileSystem   hadoop.hdfs.web.TestWebHdfsWithAuthenticationFilter   hadoop.hdfs.web.TestWebHDFSAcl   hadoop.hdfs.TestHDFSTrash   hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.TestDataTransferKeepalive   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer   hadoop.hdfs.web.TestWebHDFSForHA   hadoop.hdfs.TestBlockMissingException   hadoop.hdfs.TestPipelines   hadoop.hdfs.TestRenameWhileOpen   hadoop.hdfs.TestFileCreationClient   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.TestFileAppend3   hadoop.hdfs.TestBalancerBandwidth   hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer   hadoop.hdfs.TestSeekBug   hadoop.hdfs.TestParallelShortCircuitReadUnCached   hadoop.hdfs.TestBlockReaderLocal   hadoop.hdfs.TestListFilesInFileContext   hadoop.hdfs.web.TestWebHDFSXAttr   hadoop.hdfs.TestFileStatus   hadoop.hdfs.web.TestFSMainOperationsWebHdfs Timed out tests org.apache.hadoop.hdfs.TestFileCreation Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12750254/HDFS-8859.004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 53bef9c checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11987/artifact/patchprocess/diffcheckstylehadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11987/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/11987/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11987/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11987/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11987/console This message was automatically generated.
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        This is a good change although it does not reduce the overall datanode memory footprint much. (For 10m blocks, it only reduces 400MB memory. However, a datanode does not even have 1m blocks in practice.)

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - This is a good change although it does not reduce the overall datanode memory footprint much. (For 10m blocks, it only reduces 400MB memory. However, a datanode does not even have 1m blocks in practice.)
        Hide
        hitliuyi Yi Liu added a comment -

        Seems Jenkins has some problem and all are timeout, I randomly select 10 of them, they run successfully quickly, let me re-trigger the Jenkins.

        Show
        hitliuyi Yi Liu added a comment - Seems Jenkins has some problem and all are timeout, I randomly select 10 of them, they run successfully quickly, let me re-trigger the Jenkins.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 18m 56s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
        +1 javac 7m 45s There were no new javac warning messages.
        +1 javadoc 9m 45s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 45s The applied patch generated 6 new checkstyle issues (total was 12, now 16).
        -1 whitespace 0m 2s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 29s mvn install still works.
        +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
        +1 findbugs 4m 23s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        -1 common tests 22m 22s Tests failed in hadoop-common.
        -1 hdfs tests 173m 14s Tests failed in hadoop-hdfs.
            240m 55s  



        Reason Tests
        Failed unit tests hadoop.net.TestNetUtils
          hadoop.ha.TestZKFailoverController
        Timed out tests org.apache.hadoop.cli.TestHDFSCLI



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12750254/HDFS-8859.004.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 0a03054
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11992/artifact/patchprocess/diffcheckstylehadoop-common.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11992/artifact/patchprocess/whitespace.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/11992/artifact/patchprocess/testrun_hadoop-common.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11992/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11992/testReport/
        Java 1.7.0_55
        uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11992/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 56s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 45s There were no new javac warning messages. +1 javadoc 9m 45s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 45s The applied patch generated 6 new checkstyle issues (total was 12, now 16). -1 whitespace 0m 2s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 findbugs 4m 23s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 22m 22s Tests failed in hadoop-common. -1 hdfs tests 173m 14s Tests failed in hadoop-hdfs.     240m 55s   Reason Tests Failed unit tests hadoop.net.TestNetUtils   hadoop.ha.TestZKFailoverController Timed out tests org.apache.hadoop.cli.TestHDFSCLI Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12750254/HDFS-8859.004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0a03054 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11992/artifact/patchprocess/diffcheckstylehadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11992/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/11992/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11992/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11992/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11992/console This message was automatically generated.
        Hide
        hitliuyi Yi Liu added a comment -

        The two test failures are not related.

        Show
        hitliuyi Yi Liu added a comment - The two test failures are not related.
        Hide
        hitliuyi Yi Liu added a comment -

        Hi Tsz Wo Nicholas Sze, do you have time to help review latest patch? Does it look good to you? Thanks.

        Show
        hitliuyi Yi Liu added a comment - Hi Tsz Wo Nicholas Sze , do you have time to help review latest patch? Does it look good to you? Thanks.
        Hide
        umamaheswararao Uma Maheswara Rao G added a comment -

        Hi Yi, Thanks for the Nice work. I have put some time and reviewed the patch. Patch almost looks good.
        Please fix the following test nit.

        for (int i = 0; i < length; i++) {
        +      while (keys.contains(k = random.nextLong()));
        +      elements[i] = new TestElement(k, random.nextLong());
        +    }
        

        You may want to add keys when you find new random. Otherwise no point of having while here.

        Show
        umamaheswararao Uma Maheswara Rao G added a comment - Hi Yi, Thanks for the Nice work. I have put some time and reviewed the patch. Patch almost looks good. Please fix the following test nit. for ( int i = 0; i < length; i++) { + while (keys.contains(k = random.nextLong())); + elements[i] = new TestElement(k, random.nextLong()); + } You may want to add keys when you find new random. Otherwise no point of having while here.
        Hide
        hitliuyi Yi Liu added a comment -

        Thanks Uma for the review, let me update the patch to address your comment.

        Show
        hitliuyi Yi Liu added a comment - Thanks Uma for the review, let me update the patch to address your comment.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Hi Yi Liu, Thank you for working on this. Nice work here..
        I have another nit : LightWeightResizableGset need not override iterator as super class implementation is sufficient

        Show
        brahmareddy Brahma Reddy Battula added a comment - Hi Yi Liu , Thank you for working on this. Nice work here.. I have another nit : LightWeightResizableGset need not override iterator as super class implementation is sufficient
        Hide
        hitliuyi Yi Liu added a comment -

        Yes, Thanks Brahma for the comment, the default value of trackModification is true. I am uploading the patch to address it.

        Show
        hitliuyi Yi Liu added a comment - Yes, Thanks Brahma for the comment, the default value of trackModification is true. I am uploading the patch to address it.
        Hide
        hitliuyi Yi Liu added a comment -

        Update the patch:
        1. address Uma and Brahma 's comments.
        2. cleanup the whitespace and some checkstyle.

        Show
        hitliuyi Yi Liu added a comment - Update the patch: 1. address Uma and Brahma 's comments. 2. cleanup the whitespace and some checkstyle.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 19m 49s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
        +1 javac 8m 4s There were no new javac warning messages.
        +1 javadoc 10m 7s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 50s The applied patch generated 5 new checkstyle issues (total was 12, now 13).
        +1 whitespace 0m 2s The patch has no lines that end in whitespace.
        +1 install 1m 36s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 4m 23s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 common tests 22m 51s Tests passed in hadoop-common.
        -1 hdfs tests 77m 23s Tests failed in hadoop-hdfs.
            147m 20s  



        Reason Tests
        Timed out tests org.apache.hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport
          org.apache.hadoop.hdfs.server.namenode.TestFSImageWithSnapshot



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12762306/HDFS-8859.005.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 83e65c5
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12675/artifact/patchprocess/diffcheckstylehadoop-common.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/12675/artifact/patchprocess/testrun_hadoop-common.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12675/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12675/testReport/
        Java 1.7.0_55
        uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12675/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 49s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 8m 4s There were no new javac warning messages. +1 javadoc 10m 7s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 50s The applied patch generated 5 new checkstyle issues (total was 12, now 13). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 23s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 51s Tests passed in hadoop-common. -1 hdfs tests 77m 23s Tests failed in hadoop-hdfs.     147m 20s   Reason Tests Timed out tests org.apache.hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport   org.apache.hadoop.hdfs.server.namenode.TestFSImageWithSnapshot Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762306/HDFS-8859.005.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 83e65c5 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12675/artifact/patchprocess/diffcheckstylehadoop-common.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/12675/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12675/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12675/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12675/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 19m 50s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
        +1 javac 7m 59s There were no new javac warning messages.
        +1 javadoc 10m 15s There were no new javadoc warning messages.
        +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 53s The applied patch generated 5 new checkstyle issues (total was 12, now 13).
        +1 whitespace 0m 2s The patch has no lines that end in whitespace.
        +1 install 1m 36s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 4m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 common tests 22m 59s Tests passed in hadoop-common.
        -1 hdfs tests 162m 40s Tests failed in hadoop-hdfs.
            233m 0s  



        Reason Tests
        Failed unit tests hadoop.hdfs.server.blockmanagement.TestBlockManager



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12762306/HDFS-8859.005.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 83e65c5
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12676/artifact/patchprocess/diffcheckstylehadoop-common.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/12676/artifact/patchprocess/testrun_hadoop-common.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12676/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12676/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12676/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 50s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 59s There were no new javac warning messages. +1 javadoc 10m 15s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 53s The applied patch generated 5 new checkstyle issues (total was 12, now 13). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 4m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 59s Tests passed in hadoop-common. -1 hdfs tests 162m 40s Tests failed in hadoop-hdfs.     233m 0s   Reason Tests Failed unit tests hadoop.hdfs.server.blockmanagement.TestBlockManager Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762306/HDFS-8859.005.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 83e65c5 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12676/artifact/patchprocess/diffcheckstylehadoop-common.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/12676/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12676/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12676/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12676/console This message was automatically generated.
        Hide
        hitliuyi Yi Liu added a comment -

        The one test failure is not related.

        Show
        hitliuyi Yi Liu added a comment - The one test failure is not related.
        Hide
        umamaheswararao Uma Maheswara Rao G added a comment -

        Yi, Checkstyle comments are related. Can you please check them?

        Show
        umamaheswararao Uma Maheswara Rao G added a comment - Yi, Checkstyle comments are related. Can you please check them?
        Hide
        hitliuyi Yi Liu added a comment -

        Thanks Uma.
        There is an unused import, I will remove it in the new version of patch.

        ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java:69:29: Variable 'entries' must be private and have accessor methods.
        ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java:71:17: Variable 'hash_mask' must be private and have accessor methods.
        ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java:73:17: Variable 'size' must be private and have accessor methods.
        ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java:77:17: Variable 'modification' must be private and have accessor methods.

        Making the variables of super class 'protected' and modify them in sub classes is a natural behavior, I don't know why checkstype reports we should use private and access through methods. We always access the protected variables in the super class directly in other hadoop code.
        So I will leave these checkstyle items.

        Show
        hitliuyi Yi Liu added a comment - Thanks Uma. There is an unused import, I will remove it in the new version of patch. ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java:69:29: Variable 'entries' must be private and have accessor methods. ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java:71:17: Variable 'hash_mask' must be private and have accessor methods. ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java:73:17: Variable 'size' must be private and have accessor methods. ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java:77:17: Variable 'modification' must be private and have accessor methods. Making the variables of super class 'protected' and modify them in sub classes is a natural behavior, I don't know why checkstype reports we should use private and access through methods. We always access the protected variables in the super class directly in other hadoop code. So I will leave these checkstyle items.
        Hide
        hitliuyi Yi Liu added a comment -

        Update patch to remove unnecessary import.

        Show
        hitliuyi Yi Liu added a comment - Update patch to remove unnecessary import.
        Hide
        umamaheswararao Uma Maheswara Rao G added a comment -

        Thanks Yi, +1 on the latest patch

        Show
        umamaheswararao Uma Maheswara Rao G added a comment - Thanks Yi, +1 on the latest patch
        Hide
        hitliuyi Yi Liu added a comment -

        Thanks Uma, since the new patch only removes the unused import, based on the above Jenkins result, I will commit it shortly.

        Show
        hitliuyi Yi Liu added a comment - Thanks Uma, since the new patch only removes the unused import, based on the above Jenkins result, I will commit it shortly.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #8538 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8538/)
        HDFS-8859. Improve DataNode ReplicaMap memory footprint to save about 45%. (yliu) (yliu: rev d6fa34e014b0e2a61b24f05dd08ebe12354267fd)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightResizableGSet.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSet.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSetByHashMap.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGSet.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightCache.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightResizableGSet.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #8538 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8538/ ) HDFS-8859 . Improve DataNode ReplicaMap memory footprint to save about 45%. (yliu) (yliu: rev d6fa34e014b0e2a61b24f05dd08ebe12354267fd) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightResizableGSet.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSet.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSetByHashMap.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGSet.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightCache.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightResizableGSet.java
        Hide
        hitliuyi Yi Liu added a comment -

        Committed to trunk and branch-2, thanks Tsz Wo Nicholas Sze, Uma Maheswara Rao G, Brahma Reddy Battula for the reviews and comments!

        Show
        hitliuyi Yi Liu added a comment - Committed to trunk and branch-2, thanks Tsz Wo Nicholas Sze , Uma Maheswara Rao G , Brahma Reddy Battula for the reviews and comments!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #465 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/465/)
        HDFS-8859. Improve DataNode ReplicaMap memory footprint to save about 45%. (yliu) (yliu: rev d6fa34e014b0e2a61b24f05dd08ebe12354267fd)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGSet.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSet.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightCache.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightResizableGSet.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSetByHashMap.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightResizableGSet.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #465 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/465/ ) HDFS-8859 . Improve DataNode ReplicaMap memory footprint to save about 45%. (yliu) (yliu: rev d6fa34e014b0e2a61b24f05dd08ebe12354267fd) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGSet.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSet.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightResizableGSet.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSetByHashMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightResizableGSet.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1196 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1196/)
        HDFS-8859. Improve DataNode ReplicaMap memory footprint to save about 45%. (yliu) (yliu: rev d6fa34e014b0e2a61b24f05dd08ebe12354267fd)

        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightResizableGSet.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGSet.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSet.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSetByHashMap.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightResizableGSet.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightCache.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1196 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1196/ ) HDFS-8859 . Improve DataNode ReplicaMap memory footprint to save about 45%. (yliu) (yliu: rev d6fa34e014b0e2a61b24f05dd08ebe12354267fd) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightResizableGSet.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGSet.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSet.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSetByHashMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightResizableGSet.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightCache.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2401 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2401/)
        HDFS-8859. Improve DataNode ReplicaMap memory footprint to save about 45%. (yliu) (yliu: rev d6fa34e014b0e2a61b24f05dd08ebe12354267fd)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSet.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightResizableGSet.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSetByHashMap.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightResizableGSet.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGSet.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightCache.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2401 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2401/ ) HDFS-8859 . Improve DataNode ReplicaMap memory footprint to save about 45%. (yliu) (yliu: rev d6fa34e014b0e2a61b24f05dd08ebe12354267fd) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSet.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightResizableGSet.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSetByHashMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightResizableGSet.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGSet.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightCache.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #458 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/458/)
        HDFS-8859. Improve DataNode ReplicaMap memory footprint to save about 45%. (yliu) (yliu: rev d6fa34e014b0e2a61b24f05dd08ebe12354267fd)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightCache.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGSet.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSet.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightResizableGSet.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightResizableGSet.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSetByHashMap.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #458 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/458/ ) HDFS-8859 . Improve DataNode ReplicaMap memory footprint to save about 45%. (yliu) (yliu: rev d6fa34e014b0e2a61b24f05dd08ebe12354267fd) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightCache.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGSet.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSet.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightResizableGSet.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightResizableGSet.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSetByHashMap.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #433 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/433/)
        HDFS-8859. Improve DataNode ReplicaMap memory footprint to save about 45%. (yliu) (yliu: rev d6fa34e014b0e2a61b24f05dd08ebe12354267fd)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSet.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightResizableGSet.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightResizableGSet.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightCache.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSetByHashMap.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGSet.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #433 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/433/ ) HDFS-8859 . Improve DataNode ReplicaMap memory footprint to save about 45%. (yliu) (yliu: rev d6fa34e014b0e2a61b24f05dd08ebe12354267fd) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSet.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightResizableGSet.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightResizableGSet.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightCache.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSetByHashMap.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGSet.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2373 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2373/)
        HDFS-8859. Improve DataNode ReplicaMap memory footprint to save about 45%. (yliu) (yliu: rev d6fa34e014b0e2a61b24f05dd08ebe12354267fd)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSet.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGSet.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightCache.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSetByHashMap.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightResizableGSet.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightResizableGSet.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2373 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2373/ ) HDFS-8859 . Improve DataNode ReplicaMap memory footprint to save about 45%. (yliu) (yliu: rev d6fa34e014b0e2a61b24f05dd08ebe12354267fd) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSet.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGSet.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightCache.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GSetByHashMap.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLightWeightResizableGSet.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightResizableGSet.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java

          People

          • Assignee:
            hitliuyi Yi Liu
            Reporter:
            hitliuyi Yi Liu
          • Votes:
            0 Vote for this issue
            Watchers:
            20 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development