Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11755

Underconstruction blocks can be considered missing

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha2, 2.8.1
    • Fix Version/s: 2.9.0, 3.0.0-alpha4, 2.8.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Following sequence of events can lead to a block underconstruction being considered missing.

      • pipeline of 3 DNs, DN1->DN2->DN3
      • DN3 has a failing disk so some updates take a long time
      • Client writes entire block and is waiting for final ack
      • DN1, DN2 and DN3 have all received the block
      • DN1 is waiting for ACK from DN2 who is waiting for ACK from DN3
      • DN3 is having trouble finalizing the block due to the failing drive. It does eventually succeed but it is VERY slow at doing so.
      • DN2 times out waiting for DN3 and tears down its pieces of the pipeline, so DN1 notices and does the same. Neither DN1 nor DN2 finalized the block.
      • DN3 finally sends an IBR to the NN indicating the block has been received.
      • Drive containing the block on DN3 fails enough that the DN takes it offline and notifies NN of failed volume
      • NN removes DN3's replica from the triplets and then declares the block missing because there are no other replicas

      Seems like we shouldn't consider uncompleted blocks for replication.

      1. HDFS-11755.001.patch
        8 kB
        Nathan Roberts
      2. HDFS-11755.002.patch
        8 kB
        Nathan Roberts
      3. HDFS-11755-branch-2.8.002.patch
        8 kB
        Nathan Roberts
      4. HDFS-11755-branch-2.002.patch
        8 kB
        Nathan Roberts

        Issue Links

          Activity

          Hide
          raviprak Ravi Prakash added a comment - - edited

          Hi Nathan! Thank you for reporting the bug.

          Could you please specify the guarantees for data resiliency we can expect from HDFS. The way I see it, we have several options. To keep the discussion simple, I'll only focus on replication, although similar arguments apply to Erasure-Coding. Two of them are:
          1. Only when a file is closed by a client successfully, is data guaranteed to be resilient. Only if all 3 replicas failed at the same time, can data loss occur.
          2. Another option might be that when a client gets an ack for a packet from the datanode pipeline, the data is guaranteed to be persistent. Now all three replicas under construction need to fail at the same time for data to be lost.

          Do you know which one makes more sense? My vote would be for the latter. HDFS clients which write small amounts of data for a long time to a file, shouldn't have to close and re-open files for getting the resiliency guarantees of 3-way replication.

          Please let me know if my question doesn't make sense.

          Show
          raviprak Ravi Prakash added a comment - - edited Hi Nathan! Thank you for reporting the bug. Could you please specify the guarantees for data resiliency we can expect from HDFS. The way I see it, we have several options. To keep the discussion simple, I'll only focus on replication, although similar arguments apply to Erasure-Coding. Two of them are: 1. Only when a file is closed by a client successfully, is data guaranteed to be resilient. Only if all 3 replicas failed at the same time, can data loss occur. 2. Another option might be that when a client gets an ack for a packet from the datanode pipeline, the data is guaranteed to be persistent. Now all three replicas under construction need to fail at the same time for data to be lost. Do you know which one makes more sense? My vote would be for the latter. HDFS clients which write small amounts of data for a long time to a file, shouldn't have to close and re-open files for getting the resiliency guarantees of 3-way replication. Please let me know if my question doesn't make sense.
          Hide
          nroberts Nathan Roberts added a comment -

          v1 of trunk patch. branch 2 will require a separate patch.

          Show
          nroberts Nathan Roberts added a comment - v1 of trunk patch. branch 2 will require a separate patch.
          Hide
          nroberts Nathan Roberts added a comment -

          Do you know which one makes more sense?

          Not an expert in this area but here's my understanding. When a block is completed and the client has received the necessary acks, the client either adds another block, or completes the file. Both cause the namenode to consider the block complete, and at that point the namenode will properly maintain replication of the completed block. If the pipeline fails while writing, the client may (depends on policy configured) rebuild the pipeline to maintain the desired level of replication in the pipeline. So, while a block is mutating, it is the client that is ultimately responsible for making sure enough datanodes remain in the pipeline and in-sync with the data. Once a block is complete, it becomes the namenode's responsibility to maintain replication.

          If a client dies and fails to complete the last block, after a timeout, lease recovery will cause the file to be closed and the blocks to be properly synchronized and committed if possible.

          There is also hsync(), which applications can use to enhance the durability guarantees at the datanode (via fsync).

          Hope that helps a little.

          Show
          nroberts Nathan Roberts added a comment - Do you know which one makes more sense? Not an expert in this area but here's my understanding. When a block is completed and the client has received the necessary acks, the client either adds another block, or completes the file. Both cause the namenode to consider the block complete, and at that point the namenode will properly maintain replication of the completed block. If the pipeline fails while writing, the client may (depends on policy configured) rebuild the pipeline to maintain the desired level of replication in the pipeline. So, while a block is mutating, it is the client that is ultimately responsible for making sure enough datanodes remain in the pipeline and in-sync with the data. Once a block is complete, it becomes the namenode's responsibility to maintain replication. If a client dies and fails to complete the last block, after a timeout, lease recovery will cause the file to be closed and the blocks to be properly synchronized and committed if possible. There is also hsync(), which applications can use to enhance the durability guarantees at the datanode (via fsync). Hope that helps a little.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 15m 20s trunk passed
          +1 compile 0m 54s trunk passed
          +1 checkstyle 0m 39s trunk passed
          +1 mvnsite 1m 4s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          -1 findbugs 1m 56s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
          +1 javadoc 0m 44s trunk passed
          +1 mvninstall 1m 1s the patch passed
          +1 compile 0m 57s the patch passed
          +1 javac 0m 57s the patch passed
          -0 checkstyle 0m 38s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 193 unchanged - 3 fixed = 194 total (was 196)
          +1 mvnsite 1m 4s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 4s the patch passed
          +1 javadoc 0m 45s the patch passed
          -1 unit 73m 20s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          103m 5s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.TestFileCorruption
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-11755
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12867152/HDFS-11755.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f0af8b2a01b3 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7dd258d
          Default Java 1.8.0_121
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19362/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19362/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19362/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19362/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19362/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 15m 20s trunk passed +1 compile 0m 54s trunk passed +1 checkstyle 0m 39s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 15s trunk passed -1 findbugs 1m 56s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 0m 44s trunk passed +1 mvninstall 1m 1s the patch passed +1 compile 0m 57s the patch passed +1 javac 0m 57s the patch passed -0 checkstyle 0m 38s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 193 unchanged - 3 fixed = 194 total (was 196) +1 mvnsite 1m 4s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 4s the patch passed +1 javadoc 0m 45s the patch passed -1 unit 73m 20s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 103m 5s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestFileCorruption   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11755 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12867152/HDFS-11755.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f0af8b2a01b3 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7dd258d Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19362/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19362/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19362/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19362/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19362/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          nroberts Nathan Roberts added a comment -

          Fixed Checkstyle
          Fixed testSetReplicationWhenBatchIBR because it was expecting a setReplication() on a file with only under construction blocks to cause underReplicated counts to increase.

          Show
          nroberts Nathan Roberts added a comment - Fixed Checkstyle Fixed testSetReplicationWhenBatchIBR because it was expecting a setReplication() on a file with only under construction blocks to cause underReplicated counts to increase.
          Hide
          nroberts Nathan Roberts added a comment -

          branch-2 and branch-2.8 patches.

          Show
          nroberts Nathan Roberts added a comment - branch-2 and branch-2.8 patches.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 27s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 15m 25s trunk passed
          +1 compile 0m 49s trunk passed
          +1 checkstyle 0m 38s trunk passed
          +1 mvnsite 0m 55s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          -1 findbugs 1m 43s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
          +1 javadoc 0m 43s trunk passed
          +1 mvninstall 0m 53s the patch passed
          +1 compile 0m 47s the patch passed
          +1 javac 0m 47s the patch passed
          +1 checkstyle 0m 36s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 204 unchanged - 3 fixed = 204 total (was 207)
          +1 mvnsite 0m 54s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 54s the patch passed
          +1 javadoc 0m 39s the patch passed
          -1 unit 97m 53s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          126m 58s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-11755
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12867338/HDFS-11755.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d652be09b1c1 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6099dee
          Default Java 1.8.0_121
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19383/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19383/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19383/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19383/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 27s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 15m 25s trunk passed +1 compile 0m 49s trunk passed +1 checkstyle 0m 38s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 14s trunk passed -1 findbugs 1m 43s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 0m 43s trunk passed +1 mvninstall 0m 53s the patch passed +1 compile 0m 47s the patch passed +1 javac 0m 47s the patch passed +1 checkstyle 0m 36s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 204 unchanged - 3 fixed = 204 total (was 207) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 54s the patch passed +1 javadoc 0m 39s the patch passed -1 unit 97m 53s hadoop-hdfs in the patch failed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 126m 58s Reason Tests Failed junit tests hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11755 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12867338/HDFS-11755.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d652be09b1c1 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6099dee Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19383/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/19383/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19383/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19383/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 18m 50s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 1s The patch appears to include 2 new or modified test files.
          +1 mvninstall 9m 19s branch-2 passed
          +1 compile 0m 45s branch-2 passed with JDK v1.8.0_131
          +1 compile 0m 47s branch-2 passed with JDK v1.7.0_121
          +1 checkstyle 0m 32s branch-2 passed
          +1 mvnsite 0m 59s branch-2 passed
          +1 mvneclipse 0m 17s branch-2 passed
          +1 findbugs 2m 11s branch-2 passed
          +1 javadoc 0m 41s branch-2 passed with JDK v1.8.0_131
          +1 javadoc 0m 58s branch-2 passed with JDK v1.7.0_121
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 39s the patch passed with JDK v1.8.0_131
          +1 javac 0m 39s the patch passed
          +1 compile 0m 41s the patch passed with JDK v1.7.0_121
          +1 javac 0m 41s the patch passed
          +1 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 228 unchanged - 3 fixed = 228 total (was 231)
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 9s the patch passed
          +1 javadoc 0m 39s the patch passed with JDK v1.8.0_131
          +1 javadoc 1m 0s the patch passed with JDK v1.7.0_121
          -1 unit 63m 31s hadoop-hdfs in the patch failed with JDK v1.7.0_121.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          177m 26s



          Reason Tests
          JDK v1.8.0_131 Failed junit tests hadoop.hdfs.TestDFSClientRetries
            hadoop.hdfs.server.balancer.TestBalancerRPCDelay
          JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerRPCDelay
            hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:8515d35
          JIRA Issue HDFS-11755
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12867362/HDFS-11755-branch-2.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 2abc67060ce3 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 48b5faa
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19384/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19384/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19384/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 18m 50s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 1s The patch appears to include 2 new or modified test files. +1 mvninstall 9m 19s branch-2 passed +1 compile 0m 45s branch-2 passed with JDK v1.8.0_131 +1 compile 0m 47s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 0m 32s branch-2 passed +1 mvnsite 0m 59s branch-2 passed +1 mvneclipse 0m 17s branch-2 passed +1 findbugs 2m 11s branch-2 passed +1 javadoc 0m 41s branch-2 passed with JDK v1.8.0_131 +1 javadoc 0m 58s branch-2 passed with JDK v1.7.0_121 +1 mvninstall 0m 46s the patch passed +1 compile 0m 39s the patch passed with JDK v1.8.0_131 +1 javac 0m 39s the patch passed +1 compile 0m 41s the patch passed with JDK v1.7.0_121 +1 javac 0m 41s the patch passed +1 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 228 unchanged - 3 fixed = 228 total (was 231) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 9s the patch passed +1 javadoc 0m 39s the patch passed with JDK v1.8.0_131 +1 javadoc 1m 0s the patch passed with JDK v1.7.0_121 -1 unit 63m 31s hadoop-hdfs in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 177m 26s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.balancer.TestBalancerRPCDelay JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerRPCDelay   hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain Subsystem Report/Notes Docker Image:yetus/hadoop:8515d35 JIRA Issue HDFS-11755 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12867362/HDFS-11755-branch-2.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2abc67060ce3 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 48b5faa Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/19384/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19384/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19384/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          nroberts Nathan Roberts added a comment -

          The failing unit tests in trunk have been unstable in precommit:
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting.testMultipleVolFailuresOnNode
          org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure.testMultipleDatanodeFailure56
          The timed out test TestLeaseRecovery2 does not fail locally and has also been unstable across multiple precommit runs on this jira.

          Show
          nroberts Nathan Roberts added a comment - The failing unit tests in trunk have been unstable in precommit: org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting.testMultipleVolFailuresOnNode org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure.testMultipleDatanodeFailure56 The timed out test TestLeaseRecovery2 does not fail locally and has also been unstable across multiple precommit runs on this jira.
          Hide
          kihwal Kihwal Lee added a comment -

          +1 looks good.

          Show
          kihwal Kihwal Lee added a comment - +1 looks good.
          Hide
          kihwal Kihwal Lee added a comment -

          Thanks for fixing this, Nathan. I've committed this to trunk, branch-2 and branch-2.8.

          Show
          kihwal Kihwal Lee added a comment - Thanks for fixing this, Nathan. I've committed this to trunk, branch-2 and branch-2.8.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11719 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11719/)
          HDFS-11755. Underconstruction blocks can be considered missing. (kihwal: rev ad1e3e4d9f105fac246ce1bdae80e92e013b8ba5)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCorruption.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11719 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11719/ ) HDFS-11755 . Underconstruction blocks can be considered missing. (kihwal: rev ad1e3e4d9f105fac246ce1bdae80e92e013b8ba5) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCorruption.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          Hide
          raviprak Ravi Prakash added a comment -

          Thanks Nathan! That does help!

          Show
          raviprak Ravi Prakash added a comment - Thanks Nathan! That does help!
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          As discussed in HDFS-11445, a regression caused by HDFS-11445 is fixed by HDFS-11755. I'd like to backport HDFS-11755 into branch-2.7 as a result.

          Show
          jojochuang Wei-Chiu Chuang added a comment - As discussed in HDFS-11445 , a regression caused by HDFS-11445 is fixed by HDFS-11755 . I'd like to backport HDFS-11755 into branch-2.7 as a result.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Filed HDFS-12641 to initiate the discussion.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Filed HDFS-12641 to initiate the discussion.

            People

            • Assignee:
              nroberts Nathan Roberts
              Reporter:
              nroberts Nathan Roberts
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development