Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9589

Block files which have been hardlinked should be duplicated before the DataNode appends to the them

        Details

        • Type: Bug
        • Status: Resolved
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: 2.8.0
        • Fix Version/s: 2.8.0, 3.0.0-alpha1
        • Component/s: datanode
        • Labels:
          None
        • Target Version/s:

          Description

          Block files which have been hardlinked should be duplicated before the DataNode appends to the them. The patch for HDFS-8860 accidentally removed this code.

          1. Text File
            HDFS-9589.001.patch
            10 kB
            Colin P. McCabe

            Issue Links

            is broken by

            Improvement - An improvement or enhancement to an existing feature or task. HDFS-8860 Remove unused Replica copyOnWrite code

            • Major - Major loss of function.
            • Resolved

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              cmccabe Colin P. McCabe added a comment -

              Thanks to Lei (Eddy) Xu and Tony Wu for pointing out this error.

              Show
              cmccabe Colin P. McCabe added a comment - Thanks to Lei (Eddy) Xu and Tony Wu for pointing out this error.
              Hide
              Permalink
              eddyxu Lei (Eddy) Xu added a comment -

              +1. The changes are most brought from what deleted from HDFS-8860, with appropriate names.

              Thanks a lot for helping with this patch.

              Show
              eddyxu Lei (Eddy) Xu added a comment - +1. The changes are most brought from what deleted from HDFS-8860 , with appropriate names. Thanks a lot for helping with this patch.
              Hide
              Permalink
              hadoopqa Hadoop QA added a comment -
              -1 overall



              Vote Subsystem Runtime Comment
              0 reexec 0m 0s Docker mode activated.
              +1 @author 0m 0s The patch does not contain any @author tags.
              +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              +1 mvninstall 8m 14s trunk passed
              +1 compile 0m 45s trunk passed with JDK v1.8.0_66
              +1 compile 0m 43s trunk passed with JDK v1.7.0_91
              +1 checkstyle 0m 17s trunk passed
              +1 mvnsite 0m 56s trunk passed
              +1 mvneclipse 0m 16s trunk passed
              +1 findbugs 1m 59s trunk passed
              +1 javadoc 1m 13s trunk passed with JDK v1.8.0_66
              +1 javadoc 1m 54s trunk passed with JDK v1.7.0_91
              +1 mvninstall 0m 51s the patch passed
              +1 compile 0m 44s the patch passed with JDK v1.8.0_66
              +1 javac 0m 44s the patch passed
              +1 compile 0m 44s the patch passed with JDK v1.7.0_91
              +1 javac 0m 44s the patch passed
              -1 checkstyle 0m 17s Patch generated 2 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 145, now 146).
              +1 mvnsite 0m 55s the patch passed
              +1 mvneclipse 0m 14s the patch passed
              +1 whitespace 0m 0s Patch has no whitespace issues.
              +1 findbugs 2m 11s the patch passed
              +1 javadoc 1m 11s the patch passed with JDK v1.8.0_66
              +1 javadoc 1m 53s the patch passed with JDK v1.7.0_91
              -1 unit 57m 20s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
              -1 unit 55m 32s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
              +1 asflicense 0m 26s Patch does not generate ASF License warnings.
              141m 20s



              Reason Tests
              JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
                hadoop.hdfs.TestHFlush
                hadoop.hdfs.server.namenode.ha.TestHASafeMode
                hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
                hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
                hadoop.hdfs.server.datanode.TestBlockScanner
                hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
              JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSClientRetries
                hadoop.hdfs.server.namenode.TestFSImageWithXAttr
                hadoop.hdfs.server.namenode.TestDecommissioningStatus
                hadoop.hdfs.server.namenode.TestNNThroughputBenchmark



              Subsystem Report/Notes
              Docker Image:yetus/hadoop:0ca8df7
              JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778922/HDFS-9589.001.patch
              JIRA Issue HDFS-9589
              Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
              uname Linux 08c41fe5d735 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
              Build tool maven
              Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
              git revision trunk / a0de702
              findbugs v3.0.0
              checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13965/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
              unit https://builds.apache.org/job/PreCommit-HDFS-Build/13965/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
              unit https://builds.apache.org/job/PreCommit-HDFS-Build/13965/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
              unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13965/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13965/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
              JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13965/testReport/
              modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
              Max memory used 75MB
              Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
              Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13965/console

              This message was automatically generated.

              Show
              hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 14s trunk passed +1 compile 0m 45s trunk passed with JDK v1.8.0_66 +1 compile 0m 43s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 59s trunk passed +1 javadoc 1m 13s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 54s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 51s the patch passed +1 compile 0m 44s the patch passed with JDK v1.8.0_66 +1 javac 0m 44s the patch passed +1 compile 0m 44s the patch passed with JDK v1.7.0_91 +1 javac 0m 44s the patch passed -1 checkstyle 0m 17s Patch generated 2 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 145, now 146). +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 11s the patch passed +1 javadoc 1m 11s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 53s the patch passed with JDK v1.7.0_91 -1 unit 57m 20s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 55m 32s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 26s Patch does not generate ASF License warnings. 141m 20s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.TestHFlush   hadoop.hdfs.server.namenode.ha.TestHASafeMode   hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.namenode.TestFSImageWithXAttr   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778922/HDFS-9589.001.patch JIRA Issue HDFS-9589 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 08c41fe5d735 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a0de702 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13965/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13965/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13965/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13965/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13965/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13965/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13965/console This message was automatically generated.
              Hide
              Permalink
              cmccabe Colin P. McCabe added a comment -

              Thanks, Lei (Eddy) Xu. The test failures appear to be spurious; they all pass on my machine. Committing to 2.8...

              Show
              cmccabe Colin P. McCabe added a comment - Thanks, Lei (Eddy) Xu . The test failures appear to be spurious; they all pass on my machine. Committing to 2.8...
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-trunk-Commit #9013 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9013/)
              HDFS-9589. Block files which have been hardlinked should be duplicated (cmccabe: rev bb540ba85aa37d9fe31e640665158afe8a936230)

              • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend.java
              • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java
              • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
              • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9013 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9013/ ) HDFS-9589 . Block files which have been hardlinked should be duplicated (cmccabe: rev bb540ba85aa37d9fe31e640665158afe8a936230) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java

                People

                • Assignee:
                  cmccabe Colin P. McCabe
                  Reporter:
                  cmccabe Colin P. McCabe
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  6 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development