Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10360

DataNode may format directory and lose blocks if current/VERSION is missing

    Details

      Description

      Under certain circumstances, if the current/VERSION of a storage directory is missing, DataNode may format the storage directory even though block files are not missing.

      This is very easy to reproduce. Simply launch a HDFS cluster and create some files. Delete current/VERSION, and restart the data node.

      After the restart, the data node will format the directory and remove all existing block files:

      2016-05-03 12:57:15,387 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /data/dfs/dn/in_use.lock acquired by nodename 5314@weichiu-dn-2.vpc.cloudera.com
      2016-05-03 12:57:15,389 INFO org.apache.hadoop.hdfs.server.common.Storage: Storage directory /data/dfs/dn is not formatted for BP-787466439-172.26.24.43-1462305406642
      2016-05-03 12:57:15,389 INFO org.apache.hadoop.hdfs.server.common.Storage: Formatting ...
      2016-05-03 12:57:15,464 INFO org.apache.hadoop.hdfs.server.common.Storage: Analyzing storage directories for bpid BP-787466439-172.26.24.43-1462305406642
      2016-05-03 12:57:15,464 INFO org.apache.hadoop.hdfs.server.common.Storage: Locking is disabled for /data/dfs/dn/current/BP-787466439-172.26.24.43-1462305406642
      2016-05-03 12:57:15,465 INFO org.apache.hadoop.hdfs.server.common.Storage: Block pool storage directory /data/dfs/dn/current/BP-787466439-172.26.24.43-1462305406642 is not formatted for BP-787466439-172
      .26.24.43-1462305406642
      2016-05-03 12:57:15,465 INFO org.apache.hadoop.hdfs.server.common.Storage: Formatting ...
      2016-05-03 12:57:15,465 INFO org.apache.hadoop.hdfs.server.common.Storage: Formatting block pool BP-787466439-172.26.24.43-1462305406642 directory /data/dfs/dn/current/BP-787466439-172.26.24.43-1462305406642/current
      

      The bug is: DataNode assumes that if none of current/VERSION, previous/, previous.tmp/, removed.tmp/, finalized.tmp/ and lastcheckpoint.tmp/ exists, the storage directory contains nothing important to HDFS and decides to format it.
      https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java#L526-L545
      However, block files may still exist, and in my opinion, we should do everything possible to retain the block files.

      I have two suggestions:

      1. check if current/ directory is empty. If not, throw an InconsistentFSStateException in Storage#analyzeStorage instead of asumming its not formatted. Or,
      2. In Storage#clearDirectory, before it formats the storage directory, rename or move current/ directory. Also, log whatever is being renamed/moved.
      1. HDFS-10360.001.patch
        3 kB
        Wei-Chiu Chuang
      2. HDFS-10360.002.patch
        16 kB
        Wei-Chiu Chuang
      3. HDFS-10360.003.patch
        4 kB
        Wei-Chiu Chuang
      4. HDFS-10360.004.patch
        7 kB
        Wei-Chiu Chuang
      5. HDFS-10360.004.patch
        7 kB
        Wei-Chiu Chuang
      6. HDFS-10360.005.patch
        8 kB
        Wei-Chiu Chuang
      7. HDFS-10360.007.patch
        10 kB
        Wei-Chiu Chuang

        Issue Links

          Activity

          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Upload a proof of concept for the fix proposed in #1.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Upload a proof of concept for the fix proposed in #1.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          In addition to the fix, it should also report the error via JMX.

          Show
          jojochuang Wei-Chiu Chuang added a comment - In addition to the fix, it should also report the error via JMX.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 7s trunk passed
          +1 compile 0m 52s trunk passed with JDK v1.8.0_92
          +1 compile 0m 43s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 0m 53s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 58s trunk passed
          +1 javadoc 1m 12s trunk passed with JDK v1.8.0_92
          +1 javadoc 1m 55s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 50s the patch passed
          +1 compile 0m 50s the patch passed with JDK v1.8.0_92
          +1 javac 0m 50s the patch passed
          +1 compile 0m 41s the patch passed with JDK v1.7.0_95
          +1 javac 0m 41s the patch passed
          +1 checkstyle 0m 20s the patch passed
          +1 mvnsite 0m 53s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 12s the patch passed
          +1 javadoc 1m 12s the patch passed with JDK v1.8.0_92
          +1 javadoc 1m 55s the patch passed with JDK v1.7.0_95
          -1 unit 107m 15s hadoop-hdfs in the patch failed with JDK v1.8.0_92.
          -1 unit 102m 20s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 25s Patch does not generate ASF License warnings.
          237m 18s



          Reason Tests
          JDK v1.8.0_92 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.server.blockmanagement.TestBlockManager
            hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.TestSafeMode
            hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.TestAsyncDFSRename
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
            hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.mover.TestStorageMover
            hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs
            hadoop.hdfs.TestAsyncDFSRename



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802046/HDFS-10360.001.patch
          JIRA Issue HDFS-10360
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f88e6eb12a2b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ed54f5f
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_92 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15350/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_92.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15350/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15350/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_92.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15350/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15350/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15350/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 7s trunk passed +1 compile 0m 52s trunk passed with JDK v1.8.0_92 +1 compile 0m 43s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 58s trunk passed +1 javadoc 1m 12s trunk passed with JDK v1.8.0_92 +1 javadoc 1m 55s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 50s the patch passed +1 compile 0m 50s the patch passed with JDK v1.8.0_92 +1 javac 0m 50s the patch passed +1 compile 0m 41s the patch passed with JDK v1.7.0_95 +1 javac 0m 41s the patch passed +1 checkstyle 0m 20s the patch passed +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 12s the patch passed +1 javadoc 1m 12s the patch passed with JDK v1.8.0_92 +1 javadoc 1m 55s the patch passed with JDK v1.7.0_95 -1 unit 107m 15s hadoop-hdfs in the patch failed with JDK v1.8.0_92. -1 unit 102m 20s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 237m 18s Reason Tests JDK v1.8.0_92 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.TestSafeMode   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.TestAsyncDFSRename JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality   hadoop.hdfs.server.namenode.ha.TestBootstrapStandby   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.mover.TestStorageMover   hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs   hadoop.hdfs.TestAsyncDFSRename Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802046/HDFS-10360.001.patch JIRA Issue HDFS-10360 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f88e6eb12a2b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ed54f5f Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_92 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15350/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_92.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15350/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15350/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_92.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15350/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15350/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15350/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Rev02: add log to record what files are being deleted during a DataNode reformat operation. There's no need to add extra JMX reporting code, as long as an IOException thrown adding the volume, it is added into JMX by the existing code.

          The test failures seems unrelated. I don't have these test failures in my tree.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Rev02: add log to record what files are being deleted during a DataNode reformat operation. There's no need to add extra JMX reporting code, as long as an IOException thrown adding the volume, it is added into JMX by the existing code. The test failures seems unrelated. I don't have these test failures in my tree.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Forgot to rebase my patch... submitted a rebased patch.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Forgot to rebase my patch... submitted a rebased patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 20s Maven dependency ordering for branch
          +1 mvninstall 6m 56s trunk passed
          +1 compile 6m 11s trunk passed with JDK v1.8.0_91
          +1 compile 6m 52s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 28s trunk passed
          +1 mvnsite 1m 28s trunk passed
          +1 mvneclipse 0m 29s trunk passed
          +1 findbugs 3m 4s trunk passed
          +1 javadoc 1m 28s trunk passed with JDK v1.8.0_91
          +1 javadoc 2m 11s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 18s the patch passed
          +1 compile 5m 53s the patch passed with JDK v1.8.0_91
          +1 javac 5m 53s the patch passed
          +1 compile 6m 55s the patch passed with JDK v1.7.0_95
          +1 javac 6m 55s the patch passed
          -1 checkstyle 1m 26s root: patch generated 6 new + 142 unchanged - 1 fixed = 148 total (was 143)
          +1 mvnsite 1m 27s the patch passed
          +1 mvneclipse 0m 29s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 2m 17s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 1m 30s the patch passed with JDK v1.8.0_91
          +1 javadoc 2m 12s the patch passed with JDK v1.7.0_95
          +1 unit 60m 20s hadoop-hdfs in the patch passed with JDK v1.8.0_91.
          -1 unit 31m 33s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91.
          -1 unit 60m 34s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          -1 unit 31m 40s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 25s Patch does not generate ASF License warnings.
          242m 33s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            Invocation of toString on files in org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory() At Storage.java:in org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory() At Storage.java:[line 350]
          JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestClientRMTokens
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.TestAsyncDFSRename
            hadoop.hdfs.TestRenameWhileOpen
            hadoop.yarn.server.resourcemanager.TestContainerResourceUsage
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestClientRMTokens



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803048/HDFS-10360.002.patch
          JIRA Issue HDFS-10360
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0847d6e642f3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 411fb4b
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15399/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15399/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15399/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15399/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15399/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15399/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15399/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15399/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15399/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15399/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 20s Maven dependency ordering for branch +1 mvninstall 6m 56s trunk passed +1 compile 6m 11s trunk passed with JDK v1.8.0_91 +1 compile 6m 52s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 28s trunk passed +1 mvnsite 1m 28s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 3m 4s trunk passed +1 javadoc 1m 28s trunk passed with JDK v1.8.0_91 +1 javadoc 2m 11s trunk passed with JDK v1.7.0_95 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 18s the patch passed +1 compile 5m 53s the patch passed with JDK v1.8.0_91 +1 javac 5m 53s the patch passed +1 compile 6m 55s the patch passed with JDK v1.7.0_95 +1 javac 6m 55s the patch passed -1 checkstyle 1m 26s root: patch generated 6 new + 142 unchanged - 1 fixed = 148 total (was 143) +1 mvnsite 1m 27s the patch passed +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 2m 17s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 1m 30s the patch passed with JDK v1.8.0_91 +1 javadoc 2m 12s the patch passed with JDK v1.7.0_95 +1 unit 60m 20s hadoop-hdfs in the patch passed with JDK v1.8.0_91. -1 unit 31m 33s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91. -1 unit 60m 34s hadoop-hdfs in the patch failed with JDK v1.7.0_95. -1 unit 31m 40s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 242m 33s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Invocation of toString on files in org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory() At Storage.java:in org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory() At Storage.java: [line 350] JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.TestAsyncDFSRename   hadoop.hdfs.TestRenameWhileOpen   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803048/HDFS-10360.002.patch JIRA Issue HDFS-10360 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0847d6e642f3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 411fb4b Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15399/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15399/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/15399/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15399/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15399/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15399/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15399/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15399/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15399/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15399/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Rev05: fixed checkstyle warning, findbug warning. Also, use Java DirectoryStream to identify an empty directory without listing the content of the directory.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Rev05: fixed checkstyle warning, findbug warning. Also, use Java DirectoryStream to identify an empty directory without listing the content of the directory.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 49s trunk passed
          +1 compile 0m 39s trunk passed with JDK v1.8.0_91
          +1 compile 0m 40s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 0m 50s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 54s trunk passed
          +1 javadoc 1m 2s trunk passed with JDK v1.8.0_91
          +1 javadoc 1m 47s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 35s the patch passed with JDK v1.8.0_91
          +1 javac 0m 35s the patch passed
          +1 compile 0m 38s the patch passed with JDK v1.7.0_95
          +1 javac 0m 38s the patch passed
          +1 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: patch generated 0 new + 115 unchanged - 3 fixed = 115 total (was 118)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 8s the patch passed
          +1 javadoc 1m 1s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 42s the patch passed with JDK v1.7.0_95
          -1 unit 55m 32s hadoop-hdfs in the patch failed with JDK v1.8.0_91.
          -1 unit 54m 16s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          135m 9s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.TestFSImage
            hadoop.hdfs.TestEncryptionZonesWithKMS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804166/HDFS-10360.005.patch
          JIRA Issue HDFS-10360
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9661d7a200f0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ad94411
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15446/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15446/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15446/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15446/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15446/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15446/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 49s trunk passed +1 compile 0m 39s trunk passed with JDK v1.8.0_91 +1 compile 0m 40s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 54s trunk passed +1 javadoc 1m 2s trunk passed with JDK v1.8.0_91 +1 javadoc 1m 47s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 45s the patch passed +1 compile 0m 35s the patch passed with JDK v1.8.0_91 +1 javac 0m 35s the patch passed +1 compile 0m 38s the patch passed with JDK v1.7.0_95 +1 javac 0m 38s the patch passed +1 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: patch generated 0 new + 115 unchanged - 3 fixed = 115 total (was 118) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 8s the patch passed +1 javadoc 1m 1s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 42s the patch passed with JDK v1.7.0_95 -1 unit 55m 32s hadoop-hdfs in the patch failed with JDK v1.8.0_91. -1 unit 54m 16s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 135m 9s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.TestFSImage   hadoop.hdfs.TestEncryptionZonesWithKMS Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804166/HDFS-10360.005.patch JIRA Issue HDFS-10360 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9661d7a200f0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ad94411 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15446/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15446/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15446/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15446/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15446/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15446/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Test failures are unrelated, and tests passed in my tree.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Test failures are unrelated, and tests passed in my tree.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Hi, Wei-Chiu Chuang Thanks a lot for this finding. It exposes risk to lose data. The code LGTM.

          Would you mind to add a functional tests that starts a MiniDFSCluster with such a "corrupted" data dir on one DN, and

          • Set volFailuresTolerated
          • Make sure that the failure volumes can be detected from DN JMX and NN JMX.
          Show
          eddyxu Lei (Eddy) Xu added a comment - Hi, Wei-Chiu Chuang Thanks a lot for this finding. It exposes risk to lose data. The code LGTM. Would you mind to add a functional tests that starts a MiniDFSCluster with such a "corrupted" data dir on one DN, and Set volFailuresTolerated Make sure that the failure volumes can be detected from DN JMX and NN JMX.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Many thanks to Lei (Eddy) Xu for the comments. Here's the new patch that includes a test. This test validates that DN sees the failed volume in its JMX message, and NN also sees the same failure.

          Please review again! Thanks again!

          Show
          jojochuang Wei-Chiu Chuang added a comment - Many thanks to Lei (Eddy) Xu for the comments. Here's the new patch that includes a test. This test validates that DN sees the failed volume in its JMX message, and NN also sees the same failure. Please review again! Thanks again!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 6m 51s trunk passed
          +1 compile 0m 47s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 54s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 43s trunk passed
          +1 javadoc 1m 8s trunk passed
          +1 mvninstall 0m 55s the patch passed
          +1 compile 0m 43s the patch passed
          +1 javac 0m 43s the patch passed
          +1 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: patch generated 0 new + 126 unchanged - 3 fixed = 126 total (was 129)
          +1 mvnsite 0m 54s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 47s the patch passed
          +1 javadoc 1m 1s the patch passed
          -1 unit 57m 21s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 18s Patch does not generate ASF License warnings.
          77m 2s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestCrcCorruption



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804520/HDFS-10360.007.patch
          JIRA Issue HDFS-10360
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux dbad8962b686 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0c6726e
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15467/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15467/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15467/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15467/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 51s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 54s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 43s trunk passed +1 javadoc 1m 8s trunk passed +1 mvninstall 0m 55s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed +1 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: patch generated 0 new + 126 unchanged - 3 fixed = 126 total (was 129) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 47s the patch passed +1 javadoc 1m 1s the patch passed -1 unit 57m 21s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 77m 2s Reason Tests Failed junit tests hadoop.hdfs.TestCrcCorruption Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804520/HDFS-10360.007.patch JIRA Issue HDFS-10360 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux dbad8962b686 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0c6726e Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15467/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15467/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15467/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15467/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          +1. Thanks Wei-Chiu Chuang. Good work.

          Show
          eddyxu Lei (Eddy) Xu added a comment - +1. Thanks Wei-Chiu Chuang . Good work.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Thanks very much to Eddy for reviewing and committing this patch!

          Show
          jojochuang Wei-Chiu Chuang added a comment - Thanks very much to Eddy for reviewing and committing this patch!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #9814 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9814/)
          HDFS-10360. DataNode may format directory and lose blocks if (lei: rev cf552aa87b4c47f0c73f51f44f3bc1d267c524cf)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataStorage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9814 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9814/ ) HDFS-10360 . DataNode may format directory and lose blocks if (lei: rev cf552aa87b4c47f0c73f51f44f3bc1d267c524cf) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataStorage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java

            People

            • Assignee:
              jojochuang Wei-Chiu Chuang
              Reporter:
              jojochuang Wei-Chiu Chuang
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development