Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6663

Admin command to track file and locations from block id

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.5.0
    • Fix Version/s: 2.7.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      A dfsadmin command that allows finding out the file and the locations given a block number will be very useful in debugging production issues. It may be possible to add this feature to Fsck, instead of creating a new command.

      1. HDFS-6663-WIP.patch
        12 kB
        Chen He
      2. HDFS-6663.patch
        23 kB
        Chen He
      3. HDFS-6663-2.patch
        23 kB
        Chen He
      4. HDFS-6663-3.patch
        23 kB
        Chen He
      5. HDFS-6663-3.patch
        23 kB
        Chen He
      6. HDFS-6663-4.patch
        22 kB
        Chen He
      7. HDFS-6663-5.patch
        23 kB
        Chen He
      8. HDFS-6663-5.patch
        23 kB
        Chen He

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          36d 3h 42m 1 Chen He 15/Aug/14 20:59
          Patch Available Patch Available Resolved Resolved
          73d 23h 36m 1 Kihwal Lee 28/Oct/14 19:35
          Resolved Resolved Closed Closed
          164d 54m 1 Vinod Kumar Vavilapalli 10/Apr/15 21:29
          feng xu made changes -
          Link This issue relates to HDFS-8246 [ HDFS-8246 ]
          feng xu made changes -
          Link This issue is duplicated by HDFS-8246 [ HDFS-8246 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue is duplicated by HDFS-8246 [ HDFS-8246 ]
          Vinod Kumar Vavilapalli made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #1916 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1916/)
          HDFS-6663. Admin command to track file and locations from block id. (kihwal: rev 371a3b87ed346732ed58a4faab0c6c1db57c86ed)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1916 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1916/ ) HDFS-6663 . Admin command to track file and locations from block id. (kihwal: rev 371a3b87ed346732ed58a4faab0c6c1db57c86ed) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1941 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1941/)
          HDFS-6663. Admin command to track file and locations from block id. (kihwal: rev 371a3b87ed346732ed58a4faab0c6c1db57c86ed)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1941 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1941/ ) HDFS-6663 . Admin command to track file and locations from block id. (kihwal: rev 371a3b87ed346732ed58a4faab0c6c1db57c86ed) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #727 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/727/)
          HDFS-6663. Admin command to track file and locations from block id. (kihwal: rev 371a3b87ed346732ed58a4faab0c6c1db57c86ed)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #727 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/727/ ) HDFS-6663 . Admin command to track file and locations from block id. (kihwal: rev 371a3b87ed346732ed58a4faab0c6c1db57c86ed) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java
          Kihwal Lee made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          Kihwal Lee made changes -
          Fix Version/s 2.7.0 [ 12327584 ]
          Target Version/s 2.6.0 [ 12327181 ] 2.7.0 [ 12327584 ]
          Hide
          Kihwal Lee added a comment -

          Committed to trunk and cherry-picked to branch-2. There was a merge conflict due to the context difference in help message in branch-2.

          Show
          Kihwal Lee added a comment - Committed to trunk and cherry-picked to branch-2. There was a merge conflict due to the context difference in help message in branch-2.
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #6370 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6370/)
          HDFS-6663. Admin command to track file and locations from block id. (kihwal: rev 371a3b87ed346732ed58a4faab0c6c1db57c86ed)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6370 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6370/ ) HDFS-6663 . Admin command to track file and locations from block id. (kihwal: rev 371a3b87ed346732ed58a4faab0c6c1db57c86ed) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          Hide
          Kihwal Lee added a comment -

          +1 the latest patch looks good.

          Show
          Kihwal Lee added a comment - +1 the latest patch looks good.
          Hide
          Chen He added a comment -

          Thank you for the review, Kihwal Lee. I did these two tests on my local machine and both passed. Here are the results:
          -------------------------------------------------------
          T E S T S
          -------------------------------------------------------

          -------------------------------------------------------
          T E S T S
          -------------------------------------------------------
          Running org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.047 sec - in org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer

          Results :

          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0

          -------------------------------------------------------
          T E S T S
          -------------------------------------------------------

          -------------------------------------------------------
          T E S T S
          -------------------------------------------------------
          Running org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.108 sec - in org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl

          Results :

          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

          Show
          Chen He added a comment - Thank you for the review, Kihwal Lee . I did these two tests on my local machine and both passed. Here are the results: ------------------------------------------------------- T E S T S ------------------------------------------------------- ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.047 sec - in org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer Results : Tests run: 5, Failures: 0, Errors: 0, Skipped: 0 ------------------------------------------------------- T E S T S ------------------------------------------------------- ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.108 sec - in org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
          Hide
          Kihwal Lee added a comment -

          These are the failed test cases. PreCommit is confused....

          TestOfflineImageViewerForAcl.testWebImageViewerForAcl:173 » Socket Connection ...
          TestOfflineImageViewer.testWebImageViewer:274 » Socket Connection reset

          Show
          Kihwal Lee added a comment - These are the failed test cases. PreCommit is confused.... TestOfflineImageViewerForAcl.testWebImageViewerForAcl:173 » Socket Connection ... TestOfflineImageViewer.testWebImageViewer:274 » Socket Connection reset
          Hide
          Chen He added a comment -

          The "TestBootstrapStandby" passed on my machine:

          -------------------------------------------------------
          T E S T S
          -------------------------------------------------------

          -------------------------------------------------------
          T E S T S
          -------------------------------------------------------
          Running org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.354 sec - in org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandby

          Results :

          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0

          Show
          Chen He added a comment - The "TestBootstrapStandby" passed on my machine: ------------------------------------------------------- T E S T S ------------------------------------------------------- ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandby Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.354 sec - in org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandby Results : Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12676639/HDFS-6663-5.patch
          against trunk revision d71d40a.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandby

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8499//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8499//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12676639/HDFS-6663-5.patch against trunk revision d71d40a. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandby +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8499//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8499//console This message is automatically generated.
          Chen He made changes -
          Attachment HDFS-6663-5.patch [ 12676639 ]
          Hide
          Chen He added a comment -

          retrigger QA

          Show
          Chen He added a comment - retrigger QA
          Hide
          Chen He added a comment -

          TestDNFencingWithReplication, TestNameEditsConfigs, and TestStandbyCheckpoints passed test on my machine. The latest QA does not show any test failure. Not sure why it gives me -1.

          Show
          Chen He added a comment - TestDNFencingWithReplication, TestNameEditsConfigs, and TestStandbyCheckpoints passed test on my machine. The latest QA does not show any test failure. Not sure why it gives me -1.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12676423/HDFS-6663-5.patch
          against trunk revision 7b0f9bb.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The test build failed in hadoop-hdfs-project/hadoop-hdfs

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8481//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8481//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12676423/HDFS-6663-5.patch against trunk revision 7b0f9bb. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The test build failed in hadoop-hdfs-project/hadoop-hdfs +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8481//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8481//console This message is automatically generated.
          Chen He made changes -
          Attachment HDFS-6663-5.patch [ 12676423 ]
          Hide
          Chen He added a comment -

          Decommission status of a block contains more details. It will show a block is decomissioning or decomissioned.

          Show
          Chen He added a comment - Decommission status of a block contains more details. It will show a block is decomissioning or decomissioned.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12673457/HDFS-6663-4.patch
          against trunk revision 9b8a35a.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
          org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs
          org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
          org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8346//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8346//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12673457/HDFS-6663-4.patch against trunk revision 9b8a35a. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8346//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8346//console This message is automatically generated.
          Chen He made changes -
          Attachment HDFS-6663-4.patch [ 12673457 ]
          Chen He made changes -
          Attachment HDFS-6663-4.patch [ 12673418 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12673418/HDFS-6663-4.patch
          against trunk revision 30d56fd.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8345//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12673418/HDFS-6663-4.patch against trunk revision 30d56fd. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8345//console This message is automatically generated.
          Hide
          Chen He added a comment -

          update patch and retriger the QA.

          Show
          Chen He added a comment - update patch and retriger the QA.
          Chen He made changes -
          Attachment HDFS-6663-4.patch [ 12673418 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12667862/HDFS-6663-3.patch
          against trunk revision b02a4b4.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          org.apache.hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7989//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7989//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667862/HDFS-6663-3.patch against trunk revision b02a4b4. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7989//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7989//console This message is automatically generated.
          Chen He made changes -
          Attachment HDFS-6663-3.patch [ 12667862 ]
          Hide
          Chen He added a comment -

          retrigger QA

          Show
          Chen He added a comment - retrigger QA
          Chen He made changes -
          Attachment HDFS-6663-3.patch [ 12662500 ]
          Hide
          Chen He added a comment -

          reattach patch to trigger QA

          Show
          Chen He added a comment - reattach patch to trigger QA
          Hide
          Chen He added a comment -

          The failed test passed on my server. It related to HDFS-6624

          Show
          Chen He added a comment - The failed test passed on my server. It related to HDFS-6624
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12662225/HDFS-6663-2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS

          The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.TestValidateConfigurationSettings
          org.apache.hadoop.hdfs.server.namenode.ha.TestHAStateTransitions
          org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
          org.apache.hadoop.hdfs.server.namenode.ha.TestHAMetrics
          org.apache.hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA
          org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache
          org.apache.hadoop.hdfs.TestHDFSServerPorts

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7650//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7650//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12662225/HDFS-6663-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.TestValidateConfigurationSettings org.apache.hadoop.hdfs.server.namenode.ha.TestHAStateTransitions org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints org.apache.hadoop.hdfs.server.namenode.ha.TestHAMetrics org.apache.hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache org.apache.hadoop.hdfs.TestHDFSServerPorts +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7650//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7650//console This message is automatically generated.
          Chen He made changes -
          Affects Version/s 2.5.0 [ 12326264 ]
          Hide
          Chen He added a comment -

          update new patch that resolve the findbugs problem. The core-test failure is because of HDFS-6694

          Show
          Chen He added a comment - update new patch that resolve the findbugs problem. The core-test failure is because of HDFS-6694
          Chen He made changes -
          Attachment HDFS-6663-2.patch [ 12662225 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12662134/HDFS-6663.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover

          The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestHDFSServerPorts
          org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
          org.apache.hadoop.hdfs.server.namenode.ha.TestHAMetrics
          org.apache.hadoop.hdfs.server.namenode.ha.TestHAStateTransitions
          org.apache.hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA
          org.apache.hadoop.hdfs.server.namenode.TestValidateConfigurationSettings

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7648//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/7648//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7648//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12662134/HDFS-6663.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestHDFSServerPorts org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints org.apache.hadoop.hdfs.server.namenode.ha.TestHAMetrics org.apache.hadoop.hdfs.server.namenode.ha.TestHAStateTransitions org.apache.hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA org.apache.hadoop.hdfs.server.namenode.TestValidateConfigurationSettings +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7648//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/7648//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7648//console This message is automatically generated.
          Hide
          Chen He added a comment -

          it can also shows how many expected, live, corrupted, decommission replicas for a given blockId, also, if a replica is corrupted, it will show the datanode with corruption reason.

          Show
          Chen He added a comment - it can also shows how many expected, live, corrupted, decommission replicas for a given blockId, also, if a replica is corrupted, it will show the datanode with corruption reason.
          Chen He made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Chen He made changes -
          Attachment HDFS-6663.patch [ 12662134 ]
          Chen He made changes -
          Attachment HDFS-6663-WIP.patch [ 12661481 ]
          Hide
          Chen He added a comment -

          In progress patch, will add more test cases that include decommission and block corruption in the next version.

          Show
          Chen He added a comment - In progress patch, will add more test cases that include decommission and block corruption in the next version.
          Hide
          Chen He added a comment -

          done with the preliminary code change, right now is working on unit test.

          Show
          Chen He added a comment - done with the preliminary code change, right now is working on unit test.
          Hide
          Chen He added a comment -

          For a huge cluster, get all files' blocks information may be costly.

          Show
          Chen He added a comment - For a huge cluster, get all files' blocks information may be costly.
          Hide
          Aaron T. Myers added a comment -

          Hey Kihwal, can't this be done currently by doing something like this?

          hdfs fsck / -files -blocks -locations | grep <block ID>
          
          Show
          Aaron T. Myers added a comment - Hey Kihwal, can't this be done currently by doing something like this? hdfs fsck / -files -blocks -locations | grep <block ID>
          Hide
          Chen He added a comment -

          This is a nice feature. I can take a look, Thank you for creating this, Kihwal Lee.

          Show
          Chen He added a comment - This is a nice feature. I can take a look, Thank you for creating this, Kihwal Lee.
          Chen He made changes -
          Field Original Value New Value
          Assignee Chen He [ airbots ]
          Kihwal Lee created issue -

            People

            • Assignee:
              Chen He
              Reporter:
              Kihwal Lee
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development