Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 2.7.2, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Opening this ticket on behalf of Daryn Sharp
      HDFS-7435 introduced a more efficiently encoded block report format designed to improve performance and reduce GC load on the NN and DNs. The NN is not advertising this capability to the DNs so old-style reports are still being used.

        Issue Links

          Activity

          Hide
          jingzhao Jing Zhao added a comment -

          Looks like the issue here is that CAPABILITIES_SUPPORTED has not been correctly initialized because the enum Capability is not loaded?

          Show
          jingzhao Jing Zhao added a comment - Looks like the issue here is that CAPABILITIES_SUPPORTED has not been correctly initialized because the enum Capability is not loaded?
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Jing Zhao: Daryn has a patch and he will upload on Monday.
          I have to delete the previous comment since I copied the wrong user.

          Show
          shahrs87 Rushabh S Shah added a comment - Jing Zhao : Daryn has a patch and he will upload on Monday. I have to delete the previous comment since I copied the wrong user.
          Hide
          jingzhao Jing Zhao added a comment -

          Any update here? I can also upload a patch to fix if Daryn Sharp is busy.

          Show
          jingzhao Jing Zhao added a comment - Any update here? I can also upload a patch to fix if Daryn Sharp is busy.
          Hide
          daryn Daryn Sharp added a comment -

          Yes, the enum is lazy initialized so it wasn't advertised. Here's the patch we've been using internally to force a static init.

          Show
          daryn Daryn Sharp added a comment - Yes, the enum is lazy initialized so it wasn't advertised. Here's the patch we've been using internally to force a static init.
          Hide
          jingzhao Jing Zhao added a comment -

          Thanks for the fix, Daryn. +1 pending Jenkins.

          Show
          jingzhao Jing Zhao added a comment - Thanks for the fix, Daryn. +1 pending Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 38s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 55s There were no new javac warning messages.
          +1 javadoc 9m 49s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 22s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 19s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 4s Pre-build of native portion
          -1 hdfs tests 171m 54s Tests failed in hadoop-hdfs.
              216m 31s  



          Reason Tests
          Timed out tests org.apache.hadoop.cli.TestHDFSCLI



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12751090/HDFS-8867.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 71aedfa
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12026/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12026/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12026/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 38s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 55s There were no new javac warning messages. +1 javadoc 9m 49s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 22s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 19s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 4s Pre-build of native portion -1 hdfs tests 171m 54s Tests failed in hadoop-hdfs.     216m 31s   Reason Tests Timed out tests org.apache.hadoop.cli.TestHDFSCLI Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751090/HDFS-8867.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 71aedfa hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12026/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12026/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12026/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 0m 8s trunk compilation may be broken.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 0m 8s The patch appears to cause the build to fail.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12751090/HDFS-8867.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 7ecbfd4
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12037/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 0m 8s trunk compilation may be broken. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 0m 8s The patch appears to cause the build to fail. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751090/HDFS-8867.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 7ecbfd4 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12037/console This message was automatically generated.
          Hide
          jingzhao Jing Zhao added a comment -

          Thanks for the fix, Daryn. I've committed this to trunk, branch-2 and branch-2.7.

          Show
          jingzhao Jing Zhao added a comment - Thanks for the fix, Daryn. I've committed this to trunk, branch-2 and branch-2.7.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8322 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8322/)
          HDFS-8867. Enable optimized block reports. Contributed by Daryn Sharp. (jing9: rev f61120d964a609ae5eabeb5c4d6c9afe0a15cad8)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8322 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8322/ ) HDFS-8867 . Enable optimized block reports. Contributed by Daryn Sharp. (jing9: rev f61120d964a609ae5eabeb5c4d6c9afe0a15cad8) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8323 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8323/)
          HDFS-8867. Enable optimized block reports. Contributed by Daryn Sharp. (jing9: rev f61120d964a609ae5eabeb5c4d6c9afe0a15cad8)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8323 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8323/ ) HDFS-8867 . Enable optimized block reports. Contributed by Daryn Sharp. (jing9: rev f61120d964a609ae5eabeb5c4d6c9afe0a15cad8) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #293 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/293/)
          HDFS-8867. Enable optimized block reports. Contributed by Daryn Sharp. (jing9: rev f61120d964a609ae5eabeb5c4d6c9afe0a15cad8)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #293 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/293/ ) HDFS-8867 . Enable optimized block reports. Contributed by Daryn Sharp. (jing9: rev f61120d964a609ae5eabeb5c4d6c9afe0a15cad8) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1023 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1023/)
          HDFS-8867. Enable optimized block reports. Contributed by Daryn Sharp. (jing9: rev f61120d964a609ae5eabeb5c4d6c9afe0a15cad8)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1023 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1023/ ) HDFS-8867 . Enable optimized block reports. Contributed by Daryn Sharp. (jing9: rev f61120d964a609ae5eabeb5c4d6c9afe0a15cad8) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Hadoop-Hdfs-trunk #2220 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2220/)
          HDFS-8867. Enable optimized block reports. Contributed by Daryn Sharp. (jing9: rev f61120d964a609ae5eabeb5c4d6c9afe0a15cad8)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Hadoop-Hdfs-trunk #2220 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2220/ ) HDFS-8867 . Enable optimized block reports. Contributed by Daryn Sharp. (jing9: rev f61120d964a609ae5eabeb5c4d6c9afe0a15cad8) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java
          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #282 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/282/)
          HDFS-8867. Enable optimized block reports. Contributed by Daryn Sharp. (jing9: rev f61120d964a609ae5eabeb5c4d6c9afe0a15cad8)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #282 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/282/ ) HDFS-8867 . Enable optimized block reports. Contributed by Daryn Sharp. (jing9: rev f61120d964a609ae5eabeb5c4d6c9afe0a15cad8) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #290 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/290/)
          HDFS-8867. Enable optimized block reports. Contributed by Daryn Sharp. (jing9: rev f61120d964a609ae5eabeb5c4d6c9afe0a15cad8)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #290 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/290/ ) HDFS-8867 . Enable optimized block reports. Contributed by Daryn Sharp. (jing9: rev f61120d964a609ae5eabeb5c4d6c9afe0a15cad8) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2239 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2239/)
          HDFS-8867. Enable optimized block reports. Contributed by Daryn Sharp. (jing9: rev f61120d964a609ae5eabeb5c4d6c9afe0a15cad8)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2239 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2239/ ) HDFS-8867 . Enable optimized block reports. Contributed by Daryn Sharp. (jing9: rev f61120d964a609ae5eabeb5c4d6c9afe0a15cad8) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java

            People

            • Assignee:
              daryn Daryn Sharp
              Reporter:
              shahrs87 Rushabh S Shah
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development