Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7433

Optimize performance of DatanodeManager's node map

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha, 3.0.0-alpha1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The datanode map is currently a TreeMap. For many thousands of datanodes, tree lookups are ~10X more expensive than a HashMap. Insertions and removals are up to 100X more expensive.

      1. HDFS-7433.patch
        3 kB
        Daryn Sharp
      2. HDFS-7433.patch
        6 kB
        Daryn Sharp
      3. HDFS-7433.patch
        6 kB
        Daryn Sharp
      4. HDFS-7433.patch
        9 kB
        Daryn Sharp

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2138 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2138/)
          HDFS-7433. Optimize performance of DatanodeManager's node map. Contributed by Daryn Sharp. (kihwal: rev 7a7960be41c32f20ffec9fea811878b113da62db)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2138 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2138/ ) HDFS-7433 . Optimize performance of DatanodeManager's node map. Contributed by Daryn Sharp. (kihwal: rev 7a7960be41c32f20ffec9fea811878b113da62db) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #190 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/190/)
          HDFS-7433. Optimize performance of DatanodeManager's node map. Contributed by Daryn Sharp. (kihwal: rev 7a7960be41c32f20ffec9fea811878b113da62db)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #190 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/190/ ) HDFS-7433 . Optimize performance of DatanodeManager's node map. Contributed by Daryn Sharp. (kihwal: rev 7a7960be41c32f20ffec9fea811878b113da62db) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #180 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/180/)
          HDFS-7433. Optimize performance of DatanodeManager's node map. Contributed by Daryn Sharp. (kihwal: rev 7a7960be41c32f20ffec9fea811878b113da62db)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #180 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/180/ ) HDFS-7433 . Optimize performance of DatanodeManager's node map. Contributed by Daryn Sharp. (kihwal: rev 7a7960be41c32f20ffec9fea811878b113da62db) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2120 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2120/)
          HDFS-7433. Optimize performance of DatanodeManager's node map. Contributed by Daryn Sharp. (kihwal: rev 7a7960be41c32f20ffec9fea811878b113da62db)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2120 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2120/ ) HDFS-7433 . Optimize performance of DatanodeManager's node map. Contributed by Daryn Sharp. (kihwal: rev 7a7960be41c32f20ffec9fea811878b113da62db) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #922 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/922/)
          HDFS-7433. Optimize performance of DatanodeManager's node map. Contributed by Daryn Sharp. (kihwal: rev 7a7960be41c32f20ffec9fea811878b113da62db)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #922 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/922/ ) HDFS-7433 . Optimize performance of DatanodeManager's node map. Contributed by Daryn Sharp. (kihwal: rev 7a7960be41c32f20ffec9fea811878b113da62db) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #191 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/191/)
          HDFS-7433. Optimize performance of DatanodeManager's node map. Contributed by Daryn Sharp. (kihwal: rev 7a7960be41c32f20ffec9fea811878b113da62db)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #191 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/191/ ) HDFS-7433 . Optimize performance of DatanodeManager's node map. Contributed by Daryn Sharp. (kihwal: rev 7a7960be41c32f20ffec9fea811878b113da62db) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          kihwal Kihwal Lee added a comment -

          Filed HADOOP-11940 for the TestTraceAdmin failure.

          Show
          kihwal Kihwal Lee added a comment - Filed HADOOP-11940 for the TestTraceAdmin failure.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 38s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 30s There were no new javac warning messages.
          +1 javadoc 9m 42s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 0m 37s The applied patch generated 1288 new checkstyle issues (total was 2219, now 1807).
          -1 whitespace 0m 3s The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 37s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 4s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 native 3m 15s Pre-build of native portion
          -1 hdfs tests 165m 27s Tests failed in hadoop-hdfs.
              206m 54s  



          Reason Tests
          Failed unit tests hadoop.tracing.TestTraceAdmin



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12703937/HDFS-7433.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / cd6b26c
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10874/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/10874/artifact/patchprocess/whitespace.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10874/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10874/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10874/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 38s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 42s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 37s The applied patch generated 1288 new checkstyle issues (total was 2219, now 1807). -1 whitespace 0m 3s The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 4s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 15s Pre-build of native portion -1 hdfs tests 165m 27s Tests failed in hadoop-hdfs.     206m 54s   Reason Tests Failed unit tests hadoop.tracing.TestTraceAdmin Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12703937/HDFS-7433.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / cd6b26c checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10874/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/10874/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10874/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10874/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10874/console This message was automatically generated.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7778 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7778/)
          HDFS-7433. Optimize performance of DatanodeManager's node map. Contributed by Daryn Sharp. (kihwal: rev 7a7960be41c32f20ffec9fea811878b113da62db)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7778 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7778/ ) HDFS-7433 . Optimize performance of DatanodeManager's node map. Contributed by Daryn Sharp. (kihwal: rev 7a7960be41c32f20ffec9fea811878b113da62db) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          Hide
          kihwal Kihwal Lee added a comment -

          I've committed this to trunk and branch-2. Thanks everyone for reviewing and Daryn for fixing it.

          Show
          kihwal Kihwal Lee added a comment - I've committed this to trunk and branch-2. Thanks everyone for reviewing and Daryn for fixing it.
          Hide
          kihwal Kihwal Lee added a comment -

          Test cases failed due to run time issues except for TestTraceAdmin. When ran locally they pass without any problem.
          As for TestTraceAdmin, it is failing the same way in trunk and for many other precommit builds.

          +1 for the latest patch.

          Show
          kihwal Kihwal Lee added a comment - Test cases failed due to run time issues except for TestTraceAdmin. When ran locally they pass without any problem. As for TestTraceAdmin, it is failing the same way in trunk and for many other precommit builds. +1 for the latest patch.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 15m 0s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 47s There were no new javac warning messages.
          +1 javadoc 9m 42s There were no new javadoc warning messages.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 12s The applied patch generated 6 new checkstyle issues (total was 928, now 82).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 5s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 native 3m 19s Pre-build of native portion
          -1 hdfs tests 165m 43s Tests failed in hadoop-hdfs.
              209m 26s  



          Reason Tests
          Failed unit tests hadoop.tracing.TestTraceAdmin
            hadoop.fs.contract.hdfs.TestHDFSContractSeek
            hadoop.hdfs.server.namenode.ha.TestQuotasWithHA
            hadoop.hdfs.TestParallelShortCircuitReadUnCached



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12703937/HDFS-7433.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / bcf2890
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10873/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10873/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10873/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10873/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 0s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 47s There were no new javac warning messages. +1 javadoc 9m 42s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 12s The applied patch generated 6 new checkstyle issues (total was 928, now 82). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 5s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 19s Pre-build of native portion -1 hdfs tests 165m 43s Tests failed in hadoop-hdfs.     209m 26s   Reason Tests Failed unit tests hadoop.tracing.TestTraceAdmin   hadoop.fs.contract.hdfs.TestHDFSContractSeek   hadoop.hdfs.server.namenode.ha.TestQuotasWithHA   hadoop.hdfs.TestParallelShortCircuitReadUnCached Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12703937/HDFS-7433.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / bcf2890 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10873/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10873/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10873/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10873/console This message was automatically generated.
          Hide
          kihwal Kihwal Lee added a comment -

          The patch looks straightforward. Kicked the precommit build. It may take longer than usual since it is the BB day.

          Show
          kihwal Kihwal Lee added a comment - The patch looks straightforward. Kicked the precommit build. It may take longer than usual since it is the BB day.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12703937/HDFS-7433.patch
          against trunk revision 9d72f93.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics

          The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9959//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9959//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12703937/HDFS-7433.patch against trunk revision 9d72f93. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9959//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9959//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12703937/HDFS-7433.patch
          against trunk revision 387f271.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
          org.apache.hadoop.hdfs.server.namenode.TestFileTruncate

          The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9876//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9876//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12703937/HDFS-7433.patch against trunk revision 387f271. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA org.apache.hadoop.hdfs.server.namenode.TestFileTruncate The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9876//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9876//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12703937/HDFS-7433.patch
          against trunk revision 30c428a.

          -1 patch. Trunk compilation may be broken.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9839//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12703937/HDFS-7433.patch against trunk revision 30c428a. -1 patch . Trunk compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9839//console This message is automatically generated.
          Hide
          daryn Daryn Sharp added a comment -

          Rebased due to decomm manager rewrite. No tests, but the change is now completely trivial.

          Show
          daryn Daryn Sharp added a comment - Rebased due to decomm manager rewrite. No tests, but the change is now completely trivial.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12692603/HDFS-7433.patch
          against trunk revision 95bfd08.

          -1 patch. Trunk compilation may be broken.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9779//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12692603/HDFS-7433.patch against trunk revision 95bfd08. -1 patch . Trunk compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9779//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12692603/HDFS-7433.patch
          against trunk revision 95bfd08.

          -1 patch. Trunk compilation may be broken.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9778//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12692603/HDFS-7433.patch against trunk revision 95bfd08. -1 patch . Trunk compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9778//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12692603/HDFS-7433.patch
          against trunk revision 95bfd08.

          -1 patch. Trunk compilation may be broken.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9777//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12692603/HDFS-7433.patch against trunk revision 95bfd08. -1 patch . Trunk compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9777//console This message is automatically generated.
          Hide
          daryn Daryn Sharp added a comment -

          Quite possibly, which is why I fixed it. I suspect the mutating hashCode caused problems but I cannot get the problem to manifest again.

          Will post updated patch soon.

          Show
          daryn Daryn Sharp added a comment - Quite possibly, which is why I fixed it. I suspect the mutating hashCode caused problems but I cannot get the problem to manifest again. Will post updated patch soon.
          Hide
          kihwal Kihwal Lee added a comment -

          Daryn Sharp Do you think HDFS-7434 was the cause of the test failure?

          Show
          kihwal Kihwal Lee added a comment - Daryn Sharp Do you think HDFS-7434 was the cause of the test failure?
          Hide
          kihwal Kihwal Lee added a comment -

          The test failure needs to be further investigated. It failed after the second registration of someStorageID4762 using a different version. Up to iteration 391, the version count was fine, so it looks like the second registration somehow did not update the version count correctly or different threads are seeing a stale value.

          First registration using version1.

          2015-01-20 17:00:55,048 INFO  blockmanagement.TestDatanodeManager (TestDatanodeManager.java:testNumVersionsReportedCorrect(121))
           - Registering node storageID: someStorageID4762, version: version1, IP address: someIPsomeStorageID4762:9000
          2015-01-20 17:00:55,048 INFO  hdfs.StateChange (DatanodeManager.java:registerDatanode(924)) - BLOCK* registerDatanode: from Mock
           for DatanodeRegistration, hashCode: 1098128995 storage someStorageID4762
          2015-01-20 17:00:55,050 INFO  blockmanagement.DatanodeDescriptor (DatanodeDescriptor.java:updateHeartbeatState(386)) - Number of
           failed storage changes from 0 to 0
          2015-01-20 17:00:55,312 INFO  net.NetworkTopology (NetworkTopology.java:add(418))
           - Adding a new node: /default-rack/someIPsomeStorageID4762:9000
          

          Second registration using version2 and the failure, "Still in map: version1 has 1".

          2015-01-20 17:11:45,444 INFO  blockmanagement.TestDatanodeManager (TestDatanodeManager.java:testNumVersionsReportedCorrect(121))
           - Registering node storageID: someStorageID4762, version: version2, IP address: someIPsomeStorageID4762:9000
          2015-01-20 17:11:45,444 INFO  hdfs.StateChange (DatanodeManager.java:registerDatanode(924)) - BLOCK* registerDatanode: from Mock for
           DatanodeRegistration, hashCode: 1098128995 storage someStorageID4762
          2015-01-20 17:11:45,444 INFO  net.NetworkTopology (NetworkTopology.java:remove(487))
           - Removing a node: /default-rack/someIPsomeStorageID4762:9000
          2015-01-20 17:11:50,742 INFO  net.NetworkTopology (NetworkTopology.java:add(418))
           - Adding a new node: /default-rack/someIPsomeStorageID4762:9000
          2015-01-20 17:11:50,759 INFO  blockmanagement.TestDatanodeManager (TestDatanodeManager.java:testNumVersionsReportedCorrect(147))
           - Still in map: version1 has 1
          

          I used the same random seed of 209693745, but could not reproduce the issue. It must be a rare race.

          Show
          kihwal Kihwal Lee added a comment - The test failure needs to be further investigated. It failed after the second registration of someStorageID4762 using a different version. Up to iteration 391, the version count was fine, so it looks like the second registration somehow did not update the version count correctly or different threads are seeing a stale value. First registration using version1. 2015-01-20 17:00:55,048 INFO blockmanagement.TestDatanodeManager (TestDatanodeManager.java:testNumVersionsReportedCorrect(121)) - Registering node storageID: someStorageID4762, version: version1, IP address: someIPsomeStorageID4762:9000 2015-01-20 17:00:55,048 INFO hdfs.StateChange (DatanodeManager.java:registerDatanode(924)) - BLOCK* registerDatanode: from Mock for DatanodeRegistration, hashCode: 1098128995 storage someStorageID4762 2015-01-20 17:00:55,050 INFO blockmanagement.DatanodeDescriptor (DatanodeDescriptor.java:updateHeartbeatState(386)) - Number of failed storage changes from 0 to 0 2015-01-20 17:00:55,312 INFO net.NetworkTopology (NetworkTopology.java:add(418)) - Adding a new node: /default-rack/someIPsomeStorageID4762:9000 Second registration using version2 and the failure, "Still in map: version1 has 1". 2015-01-20 17:11:45,444 INFO blockmanagement.TestDatanodeManager (TestDatanodeManager.java:testNumVersionsReportedCorrect(121)) - Registering node storageID: someStorageID4762, version: version2, IP address: someIPsomeStorageID4762:9000 2015-01-20 17:11:45,444 INFO hdfs.StateChange (DatanodeManager.java:registerDatanode(924)) - BLOCK* registerDatanode: from Mock for DatanodeRegistration, hashCode: 1098128995 storage someStorageID4762 2015-01-20 17:11:45,444 INFO net.NetworkTopology (NetworkTopology.java:remove(487)) - Removing a node: /default-rack/someIPsomeStorageID4762:9000 2015-01-20 17:11:50,742 INFO net.NetworkTopology (NetworkTopology.java:add(418)) - Adding a new node: /default-rack/someIPsomeStorageID4762:9000 2015-01-20 17:11:50,759 INFO blockmanagement.TestDatanodeManager (TestDatanodeManager.java:testNumVersionsReportedCorrect(147)) - Still in map: version1 has 1 I used the same random seed of 209693745, but could not reproduce the issue. It must be a rare race.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12692603/HDFS-7433.patch
          against trunk revision c94c0d2.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9276//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9276//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12692603/HDFS-7433.patch against trunk revision c94c0d2. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9276//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9276//console This message is automatically generated.
          Hide
          kihwal Kihwal Lee added a comment - - edited

          The precommit did not run, so I just kicked it. Although it is a comment only change over the previous patch, a new precommit run will be nice since the last run was over a month ago.

          Show
          kihwal Kihwal Lee added a comment - - edited The precommit did not run, so I just kicked it. Although it is a comment only change over the previous patch, a new precommit run will be nice since the last run was over a month ago.
          Hide
          daryn Daryn Sharp added a comment -

          Updated comment per Ming to explain we don't wrap around.

          Show
          daryn Daryn Sharp added a comment - Updated comment per Ming to explain we don't wrap around.
          Hide
          mingma Ming Ma added a comment -

          Daryn, I agree with you, there is no need to complicate things to address this minor issue. Maybe we can add some comments in the code to explain this? Otherwise, the patch LGTM.

          Show
          mingma Ming Ma added a comment - Daryn, I agree with you, there is no need to complicate things to address this minor issue. Maybe we can add some comments in the code to explain this? Otherwise, the patch LGTM.
          Hide
          daryn Daryn Sharp added a comment -

          Ming Ma, You are correct that it's not guaranteed to check nodes.per.interval when it's at/near the end of the list. I debated adding the "complexity" to wrap around but I chose to keep it simple. Otherwise you need to track "I scanned no nodes, but I saw some in decom with the current scan id, so I'll dump the scan number and loop around for a second pass" or "I scanned fewer than the nodes.per.interval, but saw some in decom with the current scan id prior to the first scanned (lower id), so I'll bump the scan id and loop around and re-scan until I reach the first previously scanned to avoid double scans of nodes". I went the simple route, but if you feel strongly it's an issue, I'll revisit the impl.

          Show
          daryn Daryn Sharp added a comment - Ming Ma , You are correct that it's not guaranteed to check nodes.per.interval when it's at/near the end of the list. I debated adding the "complexity" to wrap around but I chose to keep it simple. Otherwise you need to track "I scanned no nodes, but I saw some in decom with the current scan id, so I'll dump the scan number and loop around for a second pass" or "I scanned fewer than the nodes.per.interval, but saw some in decom with the current scan id prior to the first scanned (lower id), so I'll bump the scan id and loop around and re-scan until I reach the first previously scanned to avoid double scans of nodes". I went the simple route, but if you feel strongly it's an issue, I'll revisit the impl.
          Hide
          mingma Ming Ma added a comment -

          Daryn, I just reread the patch. You are right, that is not an issue.

          There is one minor case in which the call to check() could be wasted. Say "dfs.namenode.decommission.nodes.per.interval" is set to 1 and you decommission only one node. The first check() will call checkDecommissionState() and set the DN's scan number to the global scan number, then it exit. The second check() will skip checkDecommissionState() call given the DN's scan is the same as the global global scan number; The third check() will do the actual check again and so on.

          Show
          mingma Ming Ma added a comment - Daryn, I just reread the patch. You are right, that is not an issue. There is one minor case in which the call to check() could be wasted. Say "dfs.namenode.decommission.nodes.per.interval" is set to 1 and you decommission only one node. The first check() will call checkDecommissionState() and set the DN's scan number to the global scan number, then it exit. The second check() will skip checkDecommissionState() call given the DN's scan is the same as the global global scan number; The third check() will do the actual check again and so on.
          Hide
          daryn Daryn Sharp added a comment -

          Ming Ma, I'm not sure I understand your scenario. When the node starts the decomm, the monitor will notice that the scan number is less than current scan number, scan the node, set the scan number to current. Now when the next cycle starts, it will skip that node because the scan number is the same as current. It's not until the monitor hits the end of the list that it bumps the scan number which will trigger the rescan of the nodes because they have a lower scan number.

          How would the continuous re-scan case occur that you describe?

          Show
          daryn Daryn Sharp added a comment - Ming Ma , I'm not sure I understand your scenario. When the node starts the decomm, the monitor will notice that the scan number is less than current scan number, scan the node, set the scan number to current. Now when the next cycle starts, it will skip that node because the scan number is the same as current. It's not until the monitor hits the end of the list that it bumps the scan number which will trigger the rescan of the nodes because they have a lower scan number. How would the continuous re-scan case occur that you describe?
          Hide
          mingma Ming Ma added a comment -

          This is a good improvement.

          Perhaps it is better call d.decommissioningStatus.setLastScan(scanNumber); for all nodes, not just decomm-in-progress nodes. Otherwise, the NN could have nodes with much smaller number than the current scan number after some time. If you decomm those nodes later, decomm check might keep checking the same set of nodes until those nodes' scan numbers are the same as the current scan number.

          Show
          mingma Ming Ma added a comment - This is a good improvement. Perhaps it is better call d.decommissioningStatus.setLastScan(scanNumber); for all nodes, not just decomm-in-progress nodes. Otherwise, the NN could have nodes with much smaller number than the current scan number after some time. If you decomm those nodes later, decomm check might keep checking the same set of nodes until those nodes' scan numbers are the same as the current scan number.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12684443/HDFS-7433.patch
          against trunk revision c732ed7.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8879//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8879//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12684443/HDFS-7433.patch against trunk revision c732ed7. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8879//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8879//console This message is automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          +1. Thanks, Daryn.

          Show
          cmccabe Colin P. McCabe added a comment - +1. Thanks, Daryn.
          Hide
          daryn Daryn Sharp added a comment -

          Removing cached xferAddr as suggested.

          Show
          daryn Daryn Sharp added a comment - Removing cached xferAddr as suggested.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Can we separate out the DatanodeId#hashCode caching part of this into another change? Although it's not a lot of lines of code, it feels kind of tricky. And as I mentioned, it doesn't really relate to this, since we're keying on Java's good old immutable String class here, not DatanodeId.

          Otherwise, I'm +1 on the TreeMap -> HashMap part.

          Show
          cmccabe Colin P. McCabe added a comment - Can we separate out the DatanodeId#hashCode caching part of this into another change? Although it's not a lot of lines of code, it feels kind of tricky. And as I mentioned, it doesn't really relate to this, since we're keying on Java's good old immutable String class here, not DatanodeId . Otherwise, I'm +1 on the TreeMap -> HashMap part.
          Hide
          daryn Daryn Sharp added a comment -

          My bad mentioning datanodeMap - juggling too many changes. DatanodeIDs are added to collections in many other places, and equality checks occur often. My more general point is mutable hashCodes are a hidden landmine which is why I filed another jira. Dynamic computation of the xfer addr (and by extension the hash) is inefficient and generates a lot of garbage.

          I'm checking out the odd test failures. They don't appear related, at least the xml parsing and class def not founds.

          Show
          daryn Daryn Sharp added a comment - My bad mentioning datanodeMap - juggling too many changes. DatanodeIDs are added to collections in many other places, and equality checks occur often. My more general point is mutable hashCodes are a hidden landmine which is why I filed another jira. Dynamic computation of the xfer addr (and by extension the hash) is inefficient and generates a lot of garbage. I'm checking out the odd test failures. They don't appear related, at least the xml parsing and class def not founds.
          Hide
          cmccabe Colin P. McCabe added a comment -

          I might be missing something here, but it looks like the map is keyed on storageID. So how does DatanodeDescriptor#hashCode enter into it? We have a map of string to DatanodeDescriptor here, seems like the only relevant hash code is String#hashCode. What am I missing?

          In datanodeDump, seems like you can avoid creating the TreeMap... just create an array of storageID strings (with keys.values().toArray or something), call sort on it, and step through it. You get O(1) lookup from the hash table for each key.

          Show
          cmccabe Colin P. McCabe added a comment - I might be missing something here, but it looks like the map is keyed on storageID. So how does DatanodeDescriptor#hashCode enter into it? We have a map of string to DatanodeDescriptor here, seems like the only relevant hash code is String#hashCode . What am I missing? In datanodeDump , seems like you can avoid creating the TreeMap... just create an array of storageID strings (with keys.values().toArray or something), call sort on it, and step through it. You get O(1) lookup from the hash table for each key.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12683349/HDFS-7433.patch
          against trunk revision 555fa2d.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestRbwSpaceReservation
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8820//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8820//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12683349/HDFS-7433.patch against trunk revision 555fa2d. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestRbwSpaceReservation org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8820//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8820//console This message is automatically generated.
          Hide
          daryn Daryn Sharp added a comment -

          The datanode map uses a tree for three reasons:

          1. Sorted listing of datanodes for ui
          2. Sorted dump of datanodes in metasave
          3. Cyclic iteration nodes during decommission

          1 & 2 are preserved by sorting only in those cases.

          3 is preserved by tracking a scan number. The decommission manager linearly scans the nodes to find those not yet checked for decommission on the last scan. Once the full list is scanned, the scan number is bumped and previously scanned nodes are rescanned.

          Further, DatanodeID#hashCode is based on a dynamic computation of the xferAddr which aggravates slow node lookups. Patch memoizes this although this pre-existing mutable hashCode is fragile. Ie. I didn't make it any worse.

          Show
          daryn Daryn Sharp added a comment - The datanode map uses a tree for three reasons: Sorted listing of datanodes for ui Sorted dump of datanodes in metasave Cyclic iteration nodes during decommission 1 & 2 are preserved by sorting only in those cases. 3 is preserved by tracking a scan number. The decommission manager linearly scans the nodes to find those not yet checked for decommission on the last scan. Once the full list is scanned, the scan number is bumped and previously scanned nodes are rescanned. Further, DatanodeID#hashCode is based on a dynamic computation of the xferAddr which aggravates slow node lookups. Patch memoizes this although this pre-existing mutable hashCode is fragile. Ie. I didn't make it any worse.

            People

            • Assignee:
              daryn Daryn Sharp
              Reporter:
              daryn Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development