Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10810

Setreplication removing block from underconstrcution temporarily when batch IBR is enabled.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      1)Batch IBR is enabled with number of committed blocks allowed=1
      2) Written one block and closed the file without waiting for IBR
      3)Setreplication called immediately on the file.

      So till the finalized IBR Received, block will not be added to neededReconstruction since following check will be false as block is not marked as complete.

      if (isNeededReconstruction(block, repl.liveReplicas())) {
              neededReconstruction.update(block, repl.liveReplicas(),
                  repl.readOnlyReplicas(), repl.decommissionedAndDecommissioning(),
                  curExpectedReplicas, curReplicasDelta, expectedReplicasDelta);
            }.

      Hence block will not marked as under-replicated.

      1. HDFS-10810-branch-2.patch
        5 kB
        Brahma Reddy Battula
      2. HDFS-10810-003.patch
        5 kB
        Brahma Reddy Battula
      3. HDFS-10810-002.patch
        4 kB
        Brahma Reddy Battula
      4. HDFS-10810.patch
        4 kB
        Brahma Reddy Battula

        Issue Links

          Activity

          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          thanks for commit and review.

          Show
          brahmareddy Brahma Reddy Battula added a comment - thanks for commit and review.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10536 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10536/)
          HDFS-10810. setreplication removing block from underconstrcution (liuml07: rev 8078a5efd0fe26b82c3768e06ccd2faddc619a7f)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCorruption.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10536 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10536/ ) HDFS-10810 . setreplication removing block from underconstrcution (liuml07: rev 8078a5efd0fe26b82c3768e06ccd2faddc619a7f) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCorruption.java
          Hide
          liuml07 Mingliang Liu added a comment -

          Committed to trunk through branch-2.8. Thanks for you contribution, Brahma Reddy Battula. Thanks for your review, Tsz Wo Nicholas Sze.

          Show
          liuml07 Mingliang Liu added a comment - Committed to trunk through branch-2.8 . Thanks for you contribution, Brahma Reddy Battula . Thanks for your review, Tsz Wo Nicholas Sze .
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          findbug and testfailure are unrelated..Bytheway will raise seperate jira for findbug.

          Show
          brahmareddy Brahma Reddy Battula added a comment - findbug and testfailure are unrelated..Bytheway will raise seperate jira for findbug.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 42s branch-2 passed
          +1 compile 0m 41s branch-2 passed with JDK v1.8.0_101
          +1 compile 0m 43s branch-2 passed with JDK v1.7.0_111
          +1 checkstyle 0m 29s branch-2 passed
          +1 mvnsite 0m 51s branch-2 passed
          +1 mvneclipse 0m 16s branch-2 passed
          -1 findbugs 1m 53s hadoop-hdfs-project/hadoop-hdfs in branch-2 has 1 extant Findbugs warnings.
          +1 javadoc 0m 54s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 1m 34s branch-2 passed with JDK v1.7.0_111
          +1 mvninstall 0m 44s the patch passed
          +1 compile 0m 36s the patch passed with JDK v1.8.0_101
          +1 javac 0m 36s the patch passed
          +1 compile 0m 40s the patch passed with JDK v1.7.0_111
          +1 javac 0m 40s the patch passed
          +1 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 160 unchanged - 1 fixed = 160 total (was 161)
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 6s the patch passed
          +1 javadoc 0m 52s the patch passed with JDK v1.8.0_101
          +1 javadoc 1m 32s the patch passed with JDK v1.7.0_111
          -1 unit 48m 50s hadoop-hdfs in the patch failed with JDK v1.7.0_111.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          126m 16s



          Reason Tests
          JDK v1.7.0_111 Failed junit tests hadoop.hdfs.TestDistributedFileSystem



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue HDFS-10810
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831266/HDFS-10810-branch-2.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 79e726f6334a 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 6dd747b
          Default Java 1.7.0_111
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/16972/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16972/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt
          JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16972/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16972/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 42s branch-2 passed +1 compile 0m 41s branch-2 passed with JDK v1.8.0_101 +1 compile 0m 43s branch-2 passed with JDK v1.7.0_111 +1 checkstyle 0m 29s branch-2 passed +1 mvnsite 0m 51s branch-2 passed +1 mvneclipse 0m 16s branch-2 passed -1 findbugs 1m 53s hadoop-hdfs-project/hadoop-hdfs in branch-2 has 1 extant Findbugs warnings. +1 javadoc 0m 54s branch-2 passed with JDK v1.8.0_101 +1 javadoc 1m 34s branch-2 passed with JDK v1.7.0_111 +1 mvninstall 0m 44s the patch passed +1 compile 0m 36s the patch passed with JDK v1.8.0_101 +1 javac 0m 36s the patch passed +1 compile 0m 40s the patch passed with JDK v1.7.0_111 +1 javac 0m 40s the patch passed +1 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 160 unchanged - 1 fixed = 160 total (was 161) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 6s the patch passed +1 javadoc 0m 52s the patch passed with JDK v1.8.0_101 +1 javadoc 1m 32s the patch passed with JDK v1.7.0_111 -1 unit 48m 50s hadoop-hdfs in the patch failed with JDK v1.7.0_111. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 126m 16s Reason Tests JDK v1.7.0_111 Failed junit tests hadoop.hdfs.TestDistributedFileSystem Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HDFS-10810 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831266/HDFS-10810-branch-2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 79e726f6334a 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 6dd747b Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/16972/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/16972/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16972/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16972/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Uploaded the branch-2 patch.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Uploaded the branch-2 patch.
          Hide
          liuml07 Mingliang Liu added a comment -

          Brahma Reddy Battula, can you confirm that the branch-2 has the same problem, and we need a branch-2 patch because of the conflict? Thanks!

          Show
          liuml07 Mingliang Liu added a comment - Brahma Reddy Battula , can you confirm that the branch-2 has the same problem, and we need a branch-2 patch because of the conflict? Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 9m 3s trunk passed
          +1 compile 0m 59s trunk passed
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 0m 54s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 49s trunk passed
          +1 javadoc 1m 1s trunk passed
          +1 mvninstall 1m 4s the patch passed
          +1 compile 0m 59s the patch passed
          +1 javac 0m 59s the patch passed
          +1 checkstyle 0m 29s the patch passed
          +1 mvnsite 1m 8s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 15s the patch passed
          +1 javadoc 1m 4s the patch passed
          -1 unit 61m 43s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 30s The patch does not generate ASF License warnings.
          85m 44s



          Reason Tests
          Failed junit tests hadoop.hdfs.web.TestWebHDFS
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.TestDFSShell



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10810
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830735/HDFS-10810-003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c108c3519360 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e19b37e
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16912/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16912/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16912/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 3s trunk passed +1 compile 0m 59s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 54s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 49s trunk passed +1 javadoc 1m 1s trunk passed +1 mvninstall 1m 4s the patch passed +1 compile 0m 59s the patch passed +1 javac 0m 59s the patch passed +1 checkstyle 0m 29s the patch passed +1 mvnsite 1m 8s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 15s the patch passed +1 javadoc 1m 4s the patch passed -1 unit 61m 43s hadoop-hdfs in the patch failed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 85m 44s Reason Tests Failed junit tests hadoop.hdfs.web.TestWebHDFS   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.TestDFSShell Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10810 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830735/HDFS-10810-003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c108c3519360 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e19b37e Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16912/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16912/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16912/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks Brahma Reddy Battula. That's true that HDFS-10666 has not addressed all the places that use fixed time sleep for waiting. It is still an in-progress JIRA and needs a lot of work. If you can file/work on new sub-tasks, I'm happy to review. The intermittently failing ones are of high priority.

          The change in the v3 patch is overall good. I'd like to confirm my thought is correct - the last getUnderReplicatedBlocksCount() and getMissingBlocksCount assertions are actually the final consistent state. That is, the UnderReplicatedBlocksCount will always be 1 and MissingBlocksCount always be 0 after they reach their respective values. If only there is a way to test this.

          Show
          liuml07 Mingliang Liu added a comment - Thanks Brahma Reddy Battula . That's true that HDFS-10666 has not addressed all the places that use fixed time sleep for waiting. It is still an in-progress JIRA and needs a lot of work. If you can file/work on new sub-tasks, I'm happy to review. The intermittently failing ones are of high priority. The change in the v3 patch is overall good. I'd like to confirm my thought is correct - the last getUnderReplicatedBlocksCount() and getMissingBlocksCount assertions are actually the final consistent state. That is, the UnderReplicatedBlocksCount will always be 1 and MissingBlocksCount always be 0 after they reach their respective values. If only there is a way to test this.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Uploaded the patch to remove the fixed time sleep..Seems HDFS-10666 not addressed all the classes,Still I can see, some classes are having fixed time sleep like TestDFSFinalize... May I can raise seperate jira's under HDFS-10666 for missed one..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Uploaded the patch to remove the fixed time sleep..Seems HDFS-10666 not addressed all the classes,Still I can see, some classes are having fixed time sleep like TestDFSFinalize ... May I can raise seperate jira's under HDFS-10666 for missed one..
          Hide
          brahmareddy Brahma Reddy Battula added a comment -
          Show
          brahmareddy Brahma Reddy Battula added a comment - Thanks Tsz Wo Nicholas Sze .
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks for your confirmation, Tsz Wo Nicholas Sze. I'll hold on commit by this week in case someone proposes concerns.

          Besides, I think this test is using fixed time sleep for event waiting. It will be great if you can use some techniques proposed in HDFS-10666, Brahma Reddy Battula.

          Show
          liuml07 Mingliang Liu added a comment - Thanks for your confirmation, Tsz Wo Nicholas Sze . I'll hold on commit by this week in case someone proposes concerns. Besides, I think this test is using fixed time sleep for event waiting. It will be great if you can use some techniques proposed in HDFS-10666 , Brahma Reddy Battula .
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Thanks for the explanation. It sounds good.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Thanks for the explanation. It sounds good.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Tsz Wo Nicholas Sze can you please take a look at this issue.?

          Show
          brahmareddy Brahma Reddy Battula added a comment - Tsz Wo Nicholas Sze can you please take a look at this issue.?
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Scenario is like

          a) Enable batchIBR and write one block and close file
          b) Call setrep on this file
          c) As block is only committed,isNeededReconstruction will be false and block will not updated in neededReconstruction hence this will not marked as underreplicated.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Scenario is like a) Enable batchIBR and write one block and close file b) Call setrep on this file c) As block is only committed, isNeededReconstruction will be false and block will not updated in neededReconstruction hence this will not marked as underreplicated.
          Hide
          liuml07 Mingliang Liu added a comment -

          Brahma Reddy Battula: we can remove v3 patch from jira..?

          I posted the v3 patch to help me understand the root cause of the problem. Sorry for the confusion. I deleted it now. Let's focus on the original v2 patch.

          Tsz Wo Nicholas Sze: Why v3 does not work?

          One change I made in deleted v3 patch was the number of DNs vs. replication factor. In the v2 test there are 3 DNs while the replication factor is set 10 from 3. I updated the v3 patch for this, and it works as what v2 expects. That is, the test fails without the patch, while it passes with the patch. Just for the record, I pasted as following instead of uploading a new patch.

            @Test(timeout = 60000)
            public void testSetReplicationWhenBatchIBR() throws Exception {
              final Configuration conf = new HdfsConfiguration();
              conf.setLong(DFSConfigKeys.DFS_BLOCKREPORT_INCREMENTAL_INTERVAL_MSEC_KEY,
                  30000);
              conf.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 1024);
              conf.setInt(DFSConfigKeys.DFS_NAMENODE_FILE_CLOSE_NUM_COMMITTED_ALLOWED_KEY,
                  1);
              try (MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf)
                  .numDataNodes(3).build()) {
                cluster.waitActive();
                final DistributedFileSystem dfs = cluster.getFileSystem();
                DFSTestUtil.createFile(dfs, new Path("/testSetReplicationWhenBatchIBR-1"),
                    1024L, (short) 3, 0L);
                //sending the FBR to Delay next IBR
                cluster.triggerBlockReports();
                Thread.sleep(3000);
                final Path filePath = new Path("/testSetReplicationWhenBatchIBR-2");
                DFSTestUtil.createFile(dfs, filePath, 1024L, (short) 3, 0L);
                dfs.setReplication(filePath, (short) 10);
                Thread.sleep(3000);
                assertEquals(0,
                    cluster.getNamesystem().getBlockManager().getMissingBlocksCount());
                assertEquals(1, cluster.getNamesystem().getBlockManager()
                    .getUnderReplicatedBlocksCount());
              }
            }
          

          Tsz Wo Nicholas Sze: I believe you have found a real bug. I just want to understand more the details.

          Me too. Again, the basic idea of taking pending replicas into account when updating the neededReconstructions makes sense to me.

          Show
          liuml07 Mingliang Liu added a comment - Brahma Reddy Battula: we can remove v3 patch from jira..? I posted the v3 patch to help me understand the root cause of the problem. Sorry for the confusion. I deleted it now. Let's focus on the original v2 patch. Tsz Wo Nicholas Sze: Why v3 does not work? One change I made in deleted v3 patch was the number of DNs vs. replication factor. In the v2 test there are 3 DNs while the replication factor is set 10 from 3. I updated the v3 patch for this, and it works as what v2 expects. That is, the test fails without the patch, while it passes with the patch. Just for the record, I pasted as following instead of uploading a new patch. @Test(timeout = 60000) public void testSetReplicationWhenBatchIBR() throws Exception { final Configuration conf = new HdfsConfiguration(); conf.setLong(DFSConfigKeys.DFS_BLOCKREPORT_INCREMENTAL_INTERVAL_MSEC_KEY, 30000); conf.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 1024); conf.setInt(DFSConfigKeys.DFS_NAMENODE_FILE_CLOSE_NUM_COMMITTED_ALLOWED_KEY, 1); try (MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf) .numDataNodes(3).build()) { cluster.waitActive(); final DistributedFileSystem dfs = cluster.getFileSystem(); DFSTestUtil.createFile(dfs, new Path( "/testSetReplicationWhenBatchIBR-1" ), 1024L, ( short ) 3, 0L); //sending the FBR to Delay next IBR cluster.triggerBlockReports(); Thread .sleep(3000); final Path filePath = new Path( "/testSetReplicationWhenBatchIBR-2" ); DFSTestUtil.createFile(dfs, filePath, 1024L, ( short ) 3, 0L); dfs.setReplication(filePath, ( short ) 10); Thread .sleep(3000); assertEquals(0, cluster.getNamesystem().getBlockManager().getMissingBlocksCount()); assertEquals(1, cluster.getNamesystem().getBlockManager() .getUnderReplicatedBlocksCount()); } } Tsz Wo Nicholas Sze: I believe you have found a real bug. I just want to understand more the details. Me too. Again, the basic idea of taking pending replicas into account when updating the neededReconstructions makes sense to me.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Brahma Reddy Battula, I believe you have found a real bug. I just want to understand more the details.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Brahma Reddy Battula , I believe you have found a real bug. I just want to understand more the details.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          The v2 and v3 patches are the same except for testSetReplicationWhenBatchIBR. Why v3 does not work?

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - The v2 and v3 patches are the same except for testSetReplicationWhenBatchIBR. Why v3 does not work?
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Can you please look at v2 patch..v3 is uploaded by Mingliang Liu and he commented here..[~liuml07] I think, we can remove v3 patch from jira..?

          Show
          brahmareddy Brahma Reddy Battula added a comment - Can you please look at v2 patch..v3 is uploaded by Mingliang Liu and he commented here ..[~liuml07] I think, we can remove v3 patch from jira..?
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Thanks for the explanation. However, the new test still failed the same way even with the fix.

          Running org.apache.hadoop.hdfs.TestFileCorruption
          Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 24.143 sec <<< FAILURE! - in org.apache.hadoop.hdfs.TestFileCorruption
          testSetReplicationWhenBatchIBR(org.apache.hadoop.hdfs.TestFileCorruption)  Time elapsed: 8.049 sec  <<< FAILURE!
          java.lang.AssertionError: expected:<1> but was:<0>
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.junit.Assert.failNotEquals(Assert.java:743)
          	at org.junit.Assert.assertEquals(Assert.java:118)
          	at org.junit.Assert.assertEquals(Assert.java:555)
          	at org.junit.Assert.assertEquals(Assert.java:542)
          	at org.apache.hadoop.hdfs.TestFileCorruption.testSetReplicationWhenBatchIBR(TestFileCorruption.java:257)
          

          I applied HDFS-10810-003.patch without any modification and ran the test a few times. It failed consistently.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Thanks for the explanation. However, the new test still failed the same way even with the fix. Running org.apache.hadoop.hdfs.TestFileCorruption Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 24.143 sec <<< FAILURE! - in org.apache.hadoop.hdfs.TestFileCorruption testSetReplicationWhenBatchIBR(org.apache.hadoop.hdfs.TestFileCorruption) Time elapsed: 8.049 sec <<< FAILURE! java.lang.AssertionError: expected:<1> but was:<0> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.hdfs.TestFileCorruption.testSetReplicationWhenBatchIBR(TestFileCorruption.java:257) I applied HDFS-10810 -003.patch without any modification and ran the test a few times. It failed consistently.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          getUnderReplicatedBlocksCount() should return 0 after calling the setreplication on that file..? am I missed something, it should return "1" right..? hence testcase failed without fix

          262	      dfs.setReplication(filePath, (short) 10);
          263	      Thread.sleep(3000);
          264	      assertEquals(1, cluster.getNamesystem().getBlockManager()
          265	          .getUnderReplicatedBlocksCount());
          
          Show
          brahmareddy Brahma Reddy Battula added a comment - getUnderReplicatedBlocksCount() should return 0 after calling the setreplication on that file..? am I missed something, it should return "1" right..? hence testcase failed without fix 262 dfs.setReplication(filePath, ( short ) 10); 263 Thread .sleep(3000); 264 assertEquals(1, cluster.getNamesystem().getBlockManager() 265 .getUnderReplicatedBlocksCount());
          Hide
          liuml07 Mingliang Liu added a comment -

          Hi Tsz Wo Nicholas Sze, please have a look at the v2 for the unit test. I think I may misunderstood what Brahma Reddy Battula meant. Thanks.

          Show
          liuml07 Mingliang Liu added a comment - Hi Tsz Wo Nicholas Sze , please have a look at the v2 for the unit test. I think I may misunderstood what Brahma Reddy Battula meant. Thanks.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          What do you expect to fail without the fix? From the new unit test, both getMissingBlocksCount() and getUnderReplicatedBlocksCount() return 0 without the fix. The behavior seems correct.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - What do you expect to fail without the fix? From the new unit test, both getMissingBlocksCount() and getUnderReplicatedBlocksCount() return 0 without the fix. The behavior seems correct.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Thanks Tsz Wo Nicholas Sze for taking look into this issue...Testcase is target for under-replicate block not for corrupt block with out fix. With fix there is chance of corrupt block hence I added that assertion.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Thanks Tsz Wo Nicholas Sze for taking look into this issue...Testcase is target for under-replicate block not for corrupt block with out fix. With fix there is chance of corrupt block hence I added that assertion.
          Hide
          brahmareddy Brahma Reddy Battula added a comment - - edited

          but I don't see why the block is counted as corrupt.

          It will mark as corrupt,after using hasEnoughEffectiveReplicas(block, repl, pendingNum,curExpectedReplicas instead of isNeededReconstruction(block, repl.liveReplicas() see the comment here for same .

          Since jira is for block is not considering for under-replicated (I did not change the jira description) ,I put underconstruction assert first so that testcase will fail without the fix.

          Show
          brahmareddy Brahma Reddy Battula added a comment - - edited but I don't see why the block is counted as corrupt. It will mark as corrupt,after using hasEnoughEffectiveReplicas(block, repl, pendingNum,curExpectedReplicas instead of isNeededReconstruction(block, repl.liveReplicas() see the comment here for same . Since jira is for block is not considering for under-replicated (I did not change the jira description) ,I put underconstruction assert first so that testcase will fail without the fix.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 26s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 10m 14s trunk passed
          +1 compile 0m 59s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 0m 57s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 54s trunk passed
          +1 javadoc 1m 3s trunk passed
          +1 mvninstall 0m 59s the patch passed
          +1 compile 0m 47s the patch passed
          +1 javac 0m 47s the patch passed
          -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 140 unchanged - 0 fixed = 141 total (was 140)
          +1 mvnsite 0m 57s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 0s the patch passed
          +1 javadoc 1m 2s the patch passed
          -1 unit 84m 47s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          109m 22s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.TestFileCorruption



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10810
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828354/HDFS-10810-003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 66bdc1dee79b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 86c9862
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16737/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16737/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16737/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16737/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 10m 14s trunk passed +1 compile 0m 59s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 54s trunk passed +1 javadoc 1m 3s trunk passed +1 mvninstall 0m 59s the patch passed +1 compile 0m 47s the patch passed +1 javac 0m 47s the patch passed -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 140 unchanged - 0 fixed = 141 total (was 140) +1 mvnsite 0m 57s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 0s the patch passed +1 javadoc 1m 2s the patch passed -1 unit 84m 47s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 109m 22s Reason Tests Failed junit tests hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestFileCorruption Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10810 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828354/HDFS-10810-003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 66bdc1dee79b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 86c9862 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16737/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16737/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16737/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16737/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Indeed, since getMissingBlocksCount() did return 0, I think the unit test cannot show the bug.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Indeed, since getMissingBlocksCount() did return 0, I think the unit test cannot show the bug.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          When I ran the new test testSetReplicationWhenBatchIBR without the fix, everything passed except that it failed at the last line TestFileCorruption.java:257 since getUnderReplicatedBlocksCount() returned 0.

          //TestFileCorruption.java:257
                assertEquals(1, cluster.getNamesystem().getBlockManager()
                    .getUnderReplicatedBlocksCount());
          
          Running org.apache.hadoop.hdfs.TestFileCorruption
          Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 25.805 sec <<< FAILURE! - in org.apache.hadoop.hdfs.TestFileCorruption
          testSetReplicationWhenBatchIBR(org.apache.hadoop.hdfs.TestFileCorruption)  Time elapsed: 9.012 sec  <<< FAILURE!
          java.lang.AssertionError: expected:<1> but was:<0>
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.junit.Assert.failNotEquals(Assert.java:743)
          	at org.junit.Assert.assertEquals(Assert.java:118)
          	at org.junit.Assert.assertEquals(Assert.java:555)
          	at org.junit.Assert.assertEquals(Assert.java:542)
          	at org.apache.hadoop.hdfs.TestFileCorruption.testSetReplicationWhenBatchIBR(TestFileCorruption.java:257)
          

          So it seems that everything works fine (or the unit test cannot show the bug)?

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - When I ran the new test testSetReplicationWhenBatchIBR without the fix, everything passed except that it failed at the last line TestFileCorruption.java:257 since getUnderReplicatedBlocksCount() returned 0. //TestFileCorruption.java:257 assertEquals(1, cluster.getNamesystem().getBlockManager() .getUnderReplicatedBlocksCount()); Running org.apache.hadoop.hdfs.TestFileCorruption Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 25.805 sec <<< FAILURE! - in org.apache.hadoop.hdfs.TestFileCorruption testSetReplicationWhenBatchIBR(org.apache.hadoop.hdfs.TestFileCorruption) Time elapsed: 9.012 sec <<< FAILURE! java.lang.AssertionError: expected:<1> but was:<0> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.hdfs.TestFileCorruption.testSetReplicationWhenBatchIBR(TestFileCorruption.java:257) So it seems that everything works fine (or the unit test cannot show the bug)?
          Hide
          liuml07 Mingliang Liu added a comment -

          Hi, Brahma Reddy Battula, I can understand that the block is removed from neededReconstruction, but I don't see why the block is counted as corrupt. I slightly simplified the unit test (correct me if I'm wrong) which mainly checks the missing block count first, but it did not fail the assertion without this patch. Also I think it's not tightly connected with batch IBR without which the IBR may still get delayed.

          The basic idea of counting pending replicas when updating the neededReconstructions makes sense to me. We need 2nd opinion here. Ping Tsz Wo Nicholas Sze. Thanks.

          Show
          liuml07 Mingliang Liu added a comment - Hi, Brahma Reddy Battula , I can understand that the block is removed from neededReconstruction , but I don't see why the block is counted as corrupt. I slightly simplified the unit test (correct me if I'm wrong) which mainly checks the missing block count first, but it did not fail the assertion without this patch. Also I think it's not tightly connected with batch IBR without which the IBR may still get delayed. The basic idea of counting pending replicas when updating the neededReconstructions makes sense to me. We need 2nd opinion here. Ping Tsz Wo Nicholas Sze . Thanks.
          Hide
          liuml07 Mingliang Liu added a comment -

          Will do this week. Thanks.

          Show
          liuml07 Mingliang Liu added a comment - Will do this week. Thanks.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Test failures are unrelated.Mingliang Liu can you look at latest patch,if you find time?

          Show
          brahmareddy Brahma Reddy Battula added a comment - Test failures are unrelated. Mingliang Liu can you look at latest patch,if you find time?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 9s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 6s trunk passed
          +1 compile 0m 44s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 43s trunk passed
          +1 javadoc 0m 53s trunk passed
          +1 mvninstall 0m 47s the patch passed
          +1 compile 0m 41s the patch passed
          +1 javac 0m 41s the patch passed
          +1 checkstyle 0m 25s the patch passed
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 46s the patch passed
          +1 javadoc 0m 53s the patch passed
          -1 unit 58m 4s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          77m 11s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestFileCreationDelete
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10810
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826171/HDFS-10810-002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 010c56d463a5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4bd45f5
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16576/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16576/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16576/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 6s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 43s trunk passed +1 javadoc 0m 53s trunk passed +1 mvninstall 0m 47s the patch passed +1 compile 0m 41s the patch passed +1 javac 0m 41s the patch passed +1 checkstyle 0m 25s the patch passed +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 53s the patch passed -1 unit 58m 4s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 77m 11s Reason Tests Failed junit tests hadoop.hdfs.TestFileCreationDelete   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10810 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826171/HDFS-10810-002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 010c56d463a5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4bd45f5 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16576/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16576/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16576/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Mingliang Liu thanks for taking a look into this issue..updated patch to check under-replicated block which was missed..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Mingliang Liu thanks for taking a look into this issue..updated patch to check under-replicated block which was missed..
          Hide
          liuml07 Mingliang Liu added a comment -

          The newly added unit test testSetReplicationWhenBatchIBR, will it fail without this fix? Thanks.

          Show
          liuml07 Mingliang Liu added a comment - The newly added unit test testSetReplicationWhenBatchIBR , will it fail without this fix? Thanks.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 47s trunk passed
          +1 compile 0m 43s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 39s trunk passed
          +1 javadoc 0m 55s trunk passed
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 141 unchanged - 0 fixed = 142 total (was 141)
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 47s the patch passed
          +1 javadoc 0m 52s the patch passed
          +1 unit 75m 55s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          94m 56s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10810
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826010/HDFS-10810.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ef3366c9aead 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c258171
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16565/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16565/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16565/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 47s trunk passed +1 compile 0m 43s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 39s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 141 unchanged - 0 fixed = 142 total (was 141) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 52s the patch passed +1 unit 75m 55s hadoop-hdfs in the patch passed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 94m 56s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10810 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826010/HDFS-10810.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ef3366c9aead 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c258171 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16565/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16565/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16565/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Without HDFS-9600 block will be marked as corrupt which added whether stored block is complete or not.

          Now isNeededReconstruction(block, repl.liveReplicas() returns false and block will be removed from the neededReconstruction

          neededReconstruction.remove(block, oldReplicas, repl.readOnlyReplicas(),
          repl.decommissionedAndDecommissioning(), oldExpectedReplicas);
          Show
          brahmareddy Brahma Reddy Battula added a comment - Without HDFS-9600 block will be marked as corrupt which added whether stored block is complete or not. Now isNeededReconstruction(block, repl.liveReplicas() returns false and block will be removed from the neededReconstruction neededReconstruction.remove(block, oldReplicas, repl.readOnlyReplicas(), repl.decommissionedAndDecommissioning(), oldExpectedReplicas);

            People

            • Assignee:
              brahmareddy Brahma Reddy Battula
              Reporter:
              brahmareddy Brahma Reddy Battula
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development