Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11708

Positional read will fail if replicas moved to different DNs after stream is opened

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Scenario:
      1. File was written to DN1, DN2 with RF=2
      2. File stream opened to read and kept. Block Locations are [DN1,DN2]
      3. One of the replica (DN2) moved to another datanode (DN3) due to datanode dead/balancing/etc.
      4. Latest block locations in NameNode will be DN1 and DN3 in the 'same order'
      5. DN1 went down, but not yet detected as dead in NameNode.
      6. Client start reading using positional read api "read(pos, buf[], offset, length)"

      1. HDFS-11708-01.patch
        12 kB
        Vinayakumar B
      2. HDFS-11708-02.patch
        16 kB
        Vinayakumar B
      3. HDFS-11708-03.patch
        16 kB
        Vinayakumar B
      4. HDFS-11708-04.patch
        16 kB
        Vinayakumar B
      5. HDFS-11708-05.patch
        16 kB
        Vinayakumar B
      6. HDFS-11708-07.patch
        22 kB
        Vinayakumar B
      7. HDFS-11708-07.patch
        22 kB
        Vinayakumar B
      8. HDFS-11708-branch-2.7-07.patch
        21 kB
        Vinayakumar B
      9. HDFS-11708-branch-2.8-07.patch
        22 kB
        Vinayakumar B
      10. HDFS-11708-branch-2-07.patch
        22 kB
        Vinayakumar B
      11. HDFS-11708-HDFS-11898-06.patch
        26 kB
        Vinayakumar B

        Issue Links

          Activity

          Hide
          vinayrpet Vinayakumar B added a comment -

          Patch for trunk

          Show
          vinayrpet Vinayakumar B added a comment - Patch for trunk
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Vinayakumar B Nice catch. two minor nits.
          1. can we fix in hedged reads also (org.apache.hadoop.hdfs.DFSInputStream#hedgedFetchBlockByteRange)?
          2. can we use GenericTestUtils instead of Thread.sleep..?

          Show
          brahmareddy Brahma Reddy Battula added a comment - Vinayakumar B Nice catch. two minor nits. 1. can we fix in hedged reads also (org.apache.hadoop.hdfs.DFSInputStream#hedgedFetchBlockByteRange)? 2. can we use GenericTestUtils instead of Thread.sleep ..?
          Hide
          vinayrpet Vinayakumar B added a comment -

          Thanks for review Brahma Reddy Battula.
          I will update the patch.

          Show
          vinayrpet Vinayakumar B added a comment - Thanks for review Brahma Reddy Battula . I will update the patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 29s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 7s Maven dependency ordering for branch
          +1 mvninstall 13m 28s trunk passed
          +1 compile 1m 24s trunk passed
          +1 checkstyle 0m 42s trunk passed
          +1 mvnsite 1m 25s trunk passed
          +1 mvneclipse 0m 29s trunk passed
          -1 findbugs 1m 19s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings.
          -1 findbugs 1m 37s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
          +1 javadoc 1m 0s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 18s the patch passed
          +1 compile 1m 22s the patch passed
          +1 javac 1m 22s the patch passed
          -0 checkstyle 0m 39s hadoop-hdfs-project: The patch generated 10 new + 182 unchanged - 0 fixed = 192 total (was 182)
          +1 mvnsite 1m 21s the patch passed
          +1 mvneclipse 0m 24s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 findbugs 3m 8s the patch passed
          +1 javadoc 0m 55s the patch passed
          +1 unit 1m 10s hadoop-hdfs-client in the patch passed.
          -1 unit 69m 19s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          103m 29s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ac17dc
          JIRA Issue HDFS-11708
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865289/HDFS-11708-01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 140605ed072a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 62579b6
          Default Java 1.8.0_121
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19213/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19213/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19213/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19213/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19213/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19213/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19213/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 29s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 13m 28s trunk passed +1 compile 1m 24s trunk passed +1 checkstyle 0m 42s trunk passed +1 mvnsite 1m 25s trunk passed +1 mvneclipse 0m 29s trunk passed -1 findbugs 1m 19s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings. -1 findbugs 1m 37s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 1m 0s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 18s the patch passed +1 compile 1m 22s the patch passed +1 javac 1m 22s the patch passed -0 checkstyle 0m 39s hadoop-hdfs-project: The patch generated 10 new + 182 unchanged - 0 fixed = 192 total (was 182) +1 mvnsite 1m 21s the patch passed +1 mvneclipse 0m 24s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 3m 8s the patch passed +1 javadoc 0m 55s the patch passed +1 unit 1m 10s hadoop-hdfs-client in the patch passed. -1 unit 69m 19s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 103m 29s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs Subsystem Report/Notes Docker Image:yetus/hadoop:0ac17dc JIRA Issue HDFS-11708 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865289/HDFS-11708-01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 140605ed072a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 62579b6 Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19213/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19213/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19213/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19213/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19213/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19213/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19213/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 36s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 7s Maven dependency ordering for branch
          +1 mvninstall 14m 1s trunk passed
          +1 compile 1m 30s trunk passed
          +1 checkstyle 0m 42s trunk passed
          +1 mvnsite 1m 36s trunk passed
          +1 mvneclipse 0m 30s trunk passed
          -1 findbugs 1m 28s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings.
          -1 findbugs 1m 52s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
          +1 javadoc 1m 10s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 1m 36s the patch passed
          +1 compile 1m 43s the patch passed
          +1 javac 1m 43s the patch passed
          -0 checkstyle 0m 44s hadoop-hdfs-project: The patch generated 10 new + 182 unchanged - 0 fixed = 192 total (was 182)
          +1 mvnsite 1m 41s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 findbugs 3m 26s the patch passed
          +1 javadoc 0m 57s the patch passed
          +1 unit 1m 12s hadoop-hdfs-client in the patch passed.
          -1 unit 64m 54s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          102m 8s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ac17dc
          JIRA Issue HDFS-11708
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865289/HDFS-11708-01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c0b6c607f138 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 62579b6
          Default Java 1.8.0_121
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19214/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19214/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19214/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19214/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19214/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19214/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19214/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 36s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 14m 1s trunk passed +1 compile 1m 30s trunk passed +1 checkstyle 0m 42s trunk passed +1 mvnsite 1m 36s trunk passed +1 mvneclipse 0m 30s trunk passed -1 findbugs 1m 28s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings. -1 findbugs 1m 52s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 1m 10s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 36s the patch passed +1 compile 1m 43s the patch passed +1 javac 1m 43s the patch passed -0 checkstyle 0m 44s hadoop-hdfs-project: The patch generated 10 new + 182 unchanged - 0 fixed = 192 total (was 182) +1 mvnsite 1m 41s the patch passed +1 mvneclipse 0m 26s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 3m 26s the patch passed +1 javadoc 0m 57s the patch passed +1 unit 1m 12s hadoop-hdfs-client in the patch passed. -1 unit 64m 54s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 102m 8s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:0ac17dc JIRA Issue HDFS-11708 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865289/HDFS-11708-01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c0b6c607f138 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 62579b6 Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19214/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19214/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19214/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19214/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19214/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19214/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19214/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Updated the patch with addressed comments.

          hedged read would pass even without these changes because there is ignoredNodes map always maintained without clearing, so there was a chance to connect to valid node and read the replica.

          Show
          vinayrpet Vinayakumar B added a comment - Updated the patch with addressed comments. hedged read would pass even without these changes because there is ignoredNodes map always maintained without clearing, so there was a chance to connect to valid node and read the replica.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 7s Maven dependency ordering for branch
          +1 mvninstall 15m 9s trunk passed
          +1 compile 1m 33s trunk passed
          +1 checkstyle 0m 47s trunk passed
          +1 mvnsite 1m 39s trunk passed
          +1 mvneclipse 0m 32s trunk passed
          -1 findbugs 1m 26s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings.
          -1 findbugs 1m 51s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
          +1 javadoc 1m 6s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 28s the patch passed
          +1 compile 1m 38s the patch passed
          +1 javac 1m 38s the patch passed
          -0 checkstyle 0m 46s hadoop-hdfs-project: The patch generated 1 new + 182 unchanged - 0 fixed = 183 total (was 182)
          +1 mvnsite 1m 28s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 findbugs 3m 25s the patch passed
          +1 javadoc 1m 1s the patch passed
          +1 unit 1m 18s hadoop-hdfs-client in the patch passed.
          -1 unit 91m 20s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          129m 29s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ac17dc
          JIRA Issue HDFS-11708
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865354/HDFS-11708-02.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 5f94e7c216a7 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 61cda39e
          Default Java 1.8.0_121
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19215/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19215/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19215/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19215/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19215/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19215/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19215/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 15m 9s trunk passed +1 compile 1m 33s trunk passed +1 checkstyle 0m 47s trunk passed +1 mvnsite 1m 39s trunk passed +1 mvneclipse 0m 32s trunk passed -1 findbugs 1m 26s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings. -1 findbugs 1m 51s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 1m 6s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 28s the patch passed +1 compile 1m 38s the patch passed +1 javac 1m 38s the patch passed -0 checkstyle 0m 46s hadoop-hdfs-project: The patch generated 1 new + 182 unchanged - 0 fixed = 183 total (was 182) +1 mvnsite 1m 28s the patch passed +1 mvneclipse 0m 25s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 3m 25s the patch passed +1 javadoc 1m 1s the patch passed +1 unit 1m 18s hadoop-hdfs-client in the patch passed. -1 unit 91m 20s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 129m 29s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:0ac17dc JIRA Issue HDFS-11708 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865354/HDFS-11708-02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5f94e7c216a7 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 61cda39e Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19215/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19215/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19215/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19215/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19215/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19215/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19215/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Updated the patch.

          Show
          vinayrpet Vinayakumar B added a comment - Updated the patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 12m 27s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 24s Maven dependency ordering for branch
          +1 mvninstall 13m 33s trunk passed
          +1 compile 1m 25s trunk passed
          +1 checkstyle 0m 41s trunk passed
          +1 mvnsite 1m 26s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          -1 findbugs 1m 21s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings.
          -1 findbugs 1m 37s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
          +1 javadoc 1m 0s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 19s the patch passed
          +1 compile 1m 23s the patch passed
          +1 javac 1m 23s the patch passed
          -0 checkstyle 0m 39s hadoop-hdfs-project: The patch generated 1 new + 182 unchanged - 0 fixed = 183 total (was 182)
          +1 mvnsite 1m 22s the patch passed
          +1 mvneclipse 0m 23s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 12s the patch passed
          +1 javadoc 0m 57s the patch passed
          +1 unit 1m 10s hadoop-hdfs-client in the patch passed.
          -1 unit 64m 33s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          111m 11s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.server.namenode.TestMetadataVersionOutput
            hadoop.hdfs.server.namenode.TestStartup



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-11708
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865874/HDFS-11708-03.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 5af24ea07487 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b0f54ea
          Default Java 1.8.0_121
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19259/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19259/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19259/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19259/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19259/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19259/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 12m 27s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 24s Maven dependency ordering for branch +1 mvninstall 13m 33s trunk passed +1 compile 1m 25s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 1m 26s trunk passed +1 mvneclipse 0m 28s trunk passed -1 findbugs 1m 21s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings. -1 findbugs 1m 37s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 1m 0s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 19s the patch passed +1 compile 1m 23s the patch passed +1 javac 1m 23s the patch passed -0 checkstyle 0m 39s hadoop-hdfs-project: The patch generated 1 new + 182 unchanged - 0 fixed = 183 total (was 182) +1 mvnsite 1m 22s the patch passed +1 mvneclipse 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 12s the patch passed +1 javadoc 0m 57s the patch passed +1 unit 1m 10s hadoop-hdfs-client in the patch passed. -1 unit 64m 33s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 111m 11s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.namenode.TestMetadataVersionOutput   hadoop.hdfs.server.namenode.TestStartup Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11708 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865874/HDFS-11708-03.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5af24ea07487 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b0f54ea Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19259/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19259/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19259/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19259/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19259/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19259/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Updated patch to fix checkstyle warning.
          Please review.

          Show
          vinayrpet Vinayakumar B added a comment - Updated patch to fix checkstyle warning. Please review.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 33s Maven dependency ordering for branch
          +1 mvninstall 13m 41s trunk passed
          +1 compile 1m 27s trunk passed
          +1 checkstyle 0m 42s trunk passed
          +1 mvnsite 1m 28s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          -1 findbugs 1m 20s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings.
          -1 findbugs 1m 39s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
          +1 javadoc 1m 1s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 20s the patch passed
          +1 compile 1m 23s the patch passed
          +1 javac 1m 23s the patch passed
          +1 checkstyle 0m 40s the patch passed
          +1 mvnsite 1m 23s the patch passed
          +1 mvneclipse 0m 23s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 16s the patch passed
          +1 javadoc 0m 59s the patch passed
          +1 unit 1m 12s hadoop-hdfs-client in the patch passed.
          -1 unit 72m 24s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          107m 41s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.server.namenode.TestMetadataVersionOutput
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.server.namenode.TestStartup



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-11708
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12866116/HDFS-11708-04.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 50f48c63e1e6 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d9014bd
          Default Java 1.8.0_121
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19291/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19291/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19291/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19291/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19291/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 33s Maven dependency ordering for branch +1 mvninstall 13m 41s trunk passed +1 compile 1m 27s trunk passed +1 checkstyle 0m 42s trunk passed +1 mvnsite 1m 28s trunk passed +1 mvneclipse 0m 28s trunk passed -1 findbugs 1m 20s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings. -1 findbugs 1m 39s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 1m 1s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 20s the patch passed +1 compile 1m 23s the patch passed +1 javac 1m 23s the patch passed +1 checkstyle 0m 40s the patch passed +1 mvnsite 1m 23s the patch passed +1 mvneclipse 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 16s the patch passed +1 javadoc 0m 59s the patch passed +1 unit 1m 12s hadoop-hdfs-client in the patch passed. -1 unit 72m 24s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 107m 41s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.namenode.TestMetadataVersionOutput   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.namenode.TestStartup Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11708 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12866116/HDFS-11708-04.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 50f48c63e1e6 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d9014bd Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19291/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19291/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/19291/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19291/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19291/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Hi Zhe Zhang,
          Can you please take a look at this?

          Show
          vinayrpet Vinayakumar B added a comment - Hi Zhe Zhang , Can you please take a look at this?
          Hide
          shv Konstantin Shvachko added a comment -

          Not sure I understand what you are trying to fix here. It looks like that chooseDataNode() already calls refreshLocatedBlock() in case when all other locations failed. So your patch adds another call to NameNode unconditionally on retry, probably not the best thing to do from performance viewpoint.
          It would be good if you could provide a unit test, that fails without your fix.

          Show
          shv Konstantin Shvachko added a comment - Not sure I understand what you are trying to fix here. It looks like that chooseDataNode() already calls refreshLocatedBlock() in case when all other locations failed. So your patch adds another call to NameNode unconditionally on retry, probably not the best thing to do from performance viewpoint. It would be good if you could provide a unit test, that fails without your fix.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Not sure I understand what you are trying to fix here. It looks like that chooseDataNode() already calls refreshLocatedBlock() in case when all other locations failed. So your patch adds another call to NameNode unconditionally on retry, probably not the best thing to do from performance viewpoint.

          I think you would have confused by the name refreshLocatedBlock(), which actually DOES NOT fetch locations from namenode if already cached for specific position. It just picks specific LocatedBlock from already fetched LocatedBlocks. Actual Namenode call to refetch happens in openInfo(true) in chooseDatanode().
          The issue was, chosen LocatedBlock was old even though cached locations were updated internally during retry in chooseDatanode(), retry was continuing with old reference of LocatedBlock, which was created outside while loop. Now on retry, it chooses again LocatedBlock from new cached locations.

          It would be good if you could provide a unit test, that fails without your fix.

          I have already included a test in patch TestPread#testPreadFailureWithChangedBlockLocations().

          Show
          vinayrpet Vinayakumar B added a comment - Not sure I understand what you are trying to fix here. It looks like that chooseDataNode() already calls refreshLocatedBlock() in case when all other locations failed. So your patch adds another call to NameNode unconditionally on retry, probably not the best thing to do from performance viewpoint. I think you would have confused by the name refreshLocatedBlock() , which actually DOES NOT fetch locations from namenode if already cached for specific position. It just picks specific LocatedBlock from already fetched LocatedBlocks. Actual Namenode call to refetch happens in openInfo(true) in chooseDatanode() . The issue was, chosen LocatedBlock was old even though cached locations were updated internally during retry in chooseDatanode() , retry was continuing with old reference of LocatedBlock, which was created outside while loop. Now on retry, it chooses again LocatedBlock from new cached locations. It would be good if you could provide a unit test, that fails without your fix. I have already included a test in patch TestPread#testPreadFailureWithChangedBlockLocations() .
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Latest patch looks good to me,thanks for updating the patch.. will hold for commit till Konstantin Shvachko check once.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Latest patch looks good to me,thanks for updating the patch.. will hold for commit till Konstantin Shvachko check once.
          Hide
          shv Konstantin Shvachko added a comment -

          Hey Vinayakumar B,

          1. Current patch does not apply cleanly to trunk any more.
          2. When I merge the test part of 004 patch, but exclude your changes to DFSInputStream, which I assume are the actual fix, then TestPread succeeds. It should fail if it captures the bug, right?
          Show
          shv Konstantin Shvachko added a comment - Hey Vinayakumar B , Current patch does not apply cleanly to trunk any more. When I merge the test part of 004 patch, but exclude your changes to DFSInputStream , which I assume are the actual fix, then TestPread succeeds. It should fail if it captures the bug, right?
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Yes, After HDFS-9807, patch will not apply cleanly. I missed this change.

          org.apache.hadoop.hdfs.TestPread#testPreadFailureWithChangedBlockLocations will fail without changes.Following is stacktrace.

          org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-586358626-10.18.246.125-1495677968099:blk_1073741825_1001 file=/test
          
          	at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:843)
          	at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:962)
          	at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1324)
          	at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1283)
          	at org.apache.hadoop.hdfs.TestPread.doPreadTestWithChangedLocations(TestPread.java:684)
          	at org.apache.hadoop.hdfs.TestPread.testPreadFailureWithChangedBlockLocations(TestPread.java:566)
          

          org.apache.hadoop.hdfs.TestPread#testPreadHedgedFailureWithChangedBlockLocations

          hedged read would pass even without these changes because there is ignoredNodes map always maintained without clearing, so there was a chance to connect to valid node and read the replica.and this addressed in HDFS-11738

          Vinayakumar B already mentioned about this.

          can we add HDFS-11738 changes here only Or can you move hedgedread changes to HDFS-11738 itself..?

          Show
          brahmareddy Brahma Reddy Battula added a comment - Yes, After HDFS-9807 , patch will not apply cleanly. I missed this change. org.apache.hadoop.hdfs.TestPread#testPreadFailureWithChangedBlockLocations will fail without changes .Following is stacktrace. org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-586358626-10.18.246.125-1495677968099:blk_1073741825_1001 file=/test at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:843) at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:962) at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1324) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1283) at org.apache.hadoop.hdfs.TestPread.doPreadTestWithChangedLocations(TestPread.java:684) at org.apache.hadoop.hdfs.TestPread.testPreadFailureWithChangedBlockLocations(TestPread.java:566) org.apache.hadoop.hdfs.TestPread#testPreadHedgedFailureWithChangedBlockLocations hedged read would pass even without these changes because there is ignoredNodes map always maintained without clearing, so there was a chance to connect to valid node and read the replica.and this addressed in HDFS-11738 Vinayakumar B already mentioned about this. can we add HDFS-11738 changes here only Or can you move hedgedread changes to HDFS-11738 itself..?
          Hide
          vinayrpet Vinayakumar B added a comment -

          Attached the rebased patch.
          TestPread.testPreadFailureWithChangedBlockLocations() fails without change.

          Show
          vinayrpet Vinayakumar B added a comment - Attached the rebased patch. TestPread.testPreadFailureWithChangedBlockLocations() fails without change.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 29s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 44s Maven dependency ordering for branch
          +1 mvninstall 15m 16s trunk passed
          +1 compile 1m 48s trunk passed
          +1 checkstyle 0m 47s trunk passed
          +1 mvnsite 1m 46s trunk passed
          +1 mvneclipse 0m 35s trunk passed
          +1 findbugs 3m 54s trunk passed
          +1 javadoc 1m 7s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 1m 41s the patch passed
          +1 compile 1m 38s the patch passed
          +1 javac 1m 38s the patch passed
          +1 checkstyle 0m 47s the patch passed
          +1 mvnsite 1m 42s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 34s the patch passed
          +1 javadoc 1m 6s the patch passed
          +1 unit 1m 21s hadoop-hdfs-client in the patch passed.
          -1 unit 96m 20s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          137m 26s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-11708
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12869785/HDFS-11708-05.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux abb01436237a 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d049bd2
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19607/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19607/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19607/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 29s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 44s Maven dependency ordering for branch +1 mvninstall 15m 16s trunk passed +1 compile 1m 48s trunk passed +1 checkstyle 0m 47s trunk passed +1 mvnsite 1m 46s trunk passed +1 mvneclipse 0m 35s trunk passed +1 findbugs 3m 54s trunk passed +1 javadoc 1m 7s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 41s the patch passed +1 compile 1m 38s the patch passed +1 javac 1m 38s the patch passed +1 checkstyle 0m 47s the patch passed +1 mvnsite 1m 42s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 34s the patch passed +1 javadoc 1m 6s the patch passed +1 unit 1m 21s hadoop-hdfs-client in the patch passed. -1 unit 96m 20s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 137m 26s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11708 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12869785/HDFS-11708-05.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux abb01436237a 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d049bd2 Default Java 1.8.0_131 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/19607/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19607/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19607/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          shv Konstantin Shvachko added a comment -

          Hey guys, I reproduced failure of TestPread#testPreadFailureWithChangedBlockLocations without the change to DFSInputStream.
          But only if I run the test case individually. If I run the entire TestPread everything passes. Could you fix this please.

          I see now that refreshLocatedBlock() uses cached locations. It seems like a rather fancy way to update a block variable that got stale. Would've preferred something more explicit, but oh well. Two observations:

          1. Should you call refreshLocatedBlock() after chooseDataNode(), not before? chooseDataNode() can update the block, but we will still work with the old locations outside.
          2. So in this case we will call refreshLocatedBlock() twice on each iteration, one inside chooseDataNode() and then one outside. Is it possible to avoid this overhead? Something like updating LocatedBlock block parameter inside chooseDataNode() by setting new values rather than re-assigning new reference, which will make block input/output parameter rather than input-only.
          Show
          shv Konstantin Shvachko added a comment - Hey guys, I reproduced failure of TestPread#testPreadFailureWithChangedBlockLocations without the change to DFSInputStream . But only if I run the test case individually. If I run the entire TestPread everything passes. Could you fix this please. I see now that refreshLocatedBlock() uses cached locations. It seems like a rather fancy way to update a block variable that got stale. Would've preferred something more explicit, but oh well. Two observations: Should you call refreshLocatedBlock() after chooseDataNode() , not before? chooseDataNode() can update the block, but we will still work with the old locations outside. So in this case we will call refreshLocatedBlock() twice on each iteration, one inside chooseDataNode() and then one outside. Is it possible to avoid this overhead? Something like updating LocatedBlock block parameter inside chooseDataNode() by setting new values rather than re-assigning new reference, which will make block input/output parameter rather than input-only.
          Hide
          vinayrpet Vinayakumar B added a comment - - edited

          Attached the patch for the update.

          1. Reason for test's failure to detect issue during full run is HDFS-11898. TestPread#testPreadFailureWithChangedBlockLocations() was also using hedged reads, for which issue will not occur. Included those changes as well here for jenkins run.
          2.

          It seems like a rather fancy way to update a block variable that got stale. Would've preferred something more explicit

          Using the return value of choseDatanode() to return the updated block as well. So avoiding the redundant calls to refreshLocatedBlock();

          Please give your thoughts.

          Show
          vinayrpet Vinayakumar B added a comment - - edited Attached the patch for the update. 1. Reason for test's failure to detect issue during full run is HDFS-11898 . TestPread#testPreadFailureWithChangedBlockLocations() was also using hedged reads, for which issue will not occur. Included those changes as well here for jenkins run. 2. It seems like a rather fancy way to update a block variable that got stale. Would've preferred something more explicit Using the return value of choseDatanode() to return the updated block as well. So avoiding the redundant calls to refreshLocatedBlock() ; Please give your thoughts.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 41s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 40s Maven dependency ordering for branch
          +1 mvninstall 16m 54s trunk passed
          +1 compile 1m 34s trunk passed
          +1 checkstyle 0m 45s trunk passed
          +1 mvnsite 1m 35s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 3m 16s trunk passed
          +1 javadoc 1m 2s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 25s the patch passed
          +1 compile 1m 24s the patch passed
          +1 javac 1m 24s the patch passed
          -0 checkstyle 0m 41s hadoop-hdfs-project: The patch generated 2 new + 232 unchanged - 0 fixed = 234 total (was 232)
          +1 mvnsite 1m 24s the patch passed
          +1 mvneclipse 0m 23s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 20s the patch passed
          +1 javadoc 0m 57s the patch passed
          +1 unit 1m 12s hadoop-hdfs-client in the patch passed.
          -1 unit 88m 42s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          128m 21s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-11708
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870241/HDFS-11708-HDFS-11898-06.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f42457a1a85c 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 31058b2
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19656/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19656/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19656/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19656/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 41s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 40s Maven dependency ordering for branch +1 mvninstall 16m 54s trunk passed +1 compile 1m 34s trunk passed +1 checkstyle 0m 45s trunk passed +1 mvnsite 1m 35s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 3m 16s trunk passed +1 javadoc 1m 2s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 25s the patch passed +1 compile 1m 24s the patch passed +1 javac 1m 24s the patch passed -0 checkstyle 0m 41s hadoop-hdfs-project: The patch generated 2 new + 232 unchanged - 0 fixed = 234 total (was 232) +1 mvnsite 1m 24s the patch passed +1 mvneclipse 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 20s the patch passed +1 javadoc 0m 57s the patch passed +1 unit 1m 12s hadoop-hdfs-client in the patch passed. -1 unit 88m 42s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 128m 21s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11708 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870241/HDFS-11708-HDFS-11898-06.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f42457a1a85c 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 31058b2 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19656/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19656/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19656/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19656/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          shv Konstantin Shvachko added a comment -

          I like the idea of returning updated block in DNAddrPair and refreshing it after every chooseTarget() call.
          Looks like you submitted two patches for both jiras here. So what do we do? Should we first commit HDFS-11898?

          Show
          shv Konstantin Shvachko added a comment - I like the idea of returning updated block in DNAddrPair and refreshing it after every chooseTarget() call. Looks like you submitted two patches for both jiras here. So what do we do? Should we first commit HDFS-11898 ?
          Hide
          vinayrpet Vinayakumar B added a comment -

          Yes. HDFS-11898 should be committed first.

          Show
          vinayrpet Vinayakumar B added a comment - Yes. HDFS-11898 should be committed first.
          Hide
          shv Konstantin Shvachko added a comment -

          Could you please attache a patch that has a single commit just for this jira.

          Show
          shv Konstantin Shvachko added a comment - Could you please attache a patch that has a single commit just for this jira.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Attached the patch for only this Jira,

          Show
          vinayrpet Vinayakumar B added a comment - Attached the patch for only this Jira,
          Hide
          vinayrpet Vinayakumar B added a comment -

          Attached patches for
          branch-2, 2.8 and 2.7.
          branch-2 and 2.8 have line differences.
          2.8 and 2.7 have some minimal refactoring differences.

          Please review

          Show
          vinayrpet Vinayakumar B added a comment - Attached patches for branch-2, 2.8 and 2.7. branch-2 and 2.8 have line differences. 2.8 and 2.7 have some minimal refactoring differences. Please review
          Hide
          shv Konstantin Shvachko added a comment -

          Latest patch HDFS-11708-07.patch is not applying cleanly to trunk. Something wrong with TestBlockReplacement.
          Other versions seem to be good to go.

          Show
          shv Konstantin Shvachko added a comment - Latest patch HDFS-11708 -07.patch is not applying cleanly to trunk. Something wrong with TestBlockReplacement. Other versions seem to be good to go.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Trunk v07 patch seems to apply cleanly for me.
          Re-attaching the same trunk patch again to verify by jenkins.

          Show
          vinayrpet Vinayakumar B added a comment - Trunk v07 patch seems to apply cleanly for me. Re-attaching the same trunk patch again to verify by jenkins.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Jenkins says v07 patch for trunk applies cleanly as well.

          ============================================================================
          ============================================================================
                                  Confirming git environment
          ============================================================================
          ============================================================================
          
          
          HEAD is now at 66c6fd8 HADOOP-14431. ModifyTime of FileStatus returned by SFTPFileSystem's getFileStatus method is wrong. Contributed by Hongyuan Li.
          Previous HEAD position was 66c6fd8... HADOOP-14431. ModifyTime of FileStatus returned by SFTPFileSystem's getFileStatus method is wrong. Contributed by Hongyuan Li.
          Switched to branch 'trunk'
          Your branch is behind 'origin/trunk' by 1 commit, and can be fast-forwarded.
            (use "git pull" to update your local branch)
          First, rewinding head to replay your work on top of it...
          Fast-forwarded trunk to 66c6fd831497944f4f49c5ce42c69a302b7d7bc0.
          Already on 'trunk'
          Your branch is up-to-date with 'origin/trunk'.
          Current branch trunk is up to date.
          Testing HDFS-11708 patch on trunk.
          
          
          ============================================================================
          ============================================================================
                                    Re-execing under Docker
          ============================================================================
          ============================================================================
          Show
          vinayrpet Vinayakumar B added a comment - Jenkins says v07 patch for trunk applies cleanly as well. ============================================================================ ============================================================================ Confirming git environment ============================================================================ ============================================================================ HEAD is now at 66c6fd8 HADOOP-14431. ModifyTime of FileStatus returned by SFTPFileSystem's getFileStatus method is wrong. Contributed by Hongyuan Li. Previous HEAD position was 66c6fd8... HADOOP-14431. ModifyTime of FileStatus returned by SFTPFileSystem's getFileStatus method is wrong. Contributed by Hongyuan Li. Switched to branch 'trunk' Your branch is behind 'origin/trunk' by 1 commit, and can be fast-forwarded. (use "git pull" to update your local branch) First, rewinding head to replay your work on top of it... Fast-forwarded trunk to 66c6fd831497944f4f49c5ce42c69a302b7d7bc0. Already on 'trunk' Your branch is up-to-date with 'origin/trunk'. Current branch trunk is up to date. Testing HDFS-11708 patch on trunk. ============================================================================ ============================================================================ Re-execing under Docker ============================================================================ ============================================================================
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 8s Maven dependency ordering for branch
          +1 mvninstall 13m 52s trunk passed
          +1 compile 1m 31s trunk passed
          +1 checkstyle 0m 44s trunk passed
          +1 mvnsite 1m 31s trunk passed
          +1 findbugs 3m 0s trunk passed
          +1 javadoc 1m 1s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 20s the patch passed
          +1 compile 1m 21s the patch passed
          +1 javac 1m 21s the patch passed
          -0 checkstyle 0m 40s hadoop-hdfs-project: The patch generated 2 new + 182 unchanged - 0 fixed = 184 total (was 182)
          +1 mvnsite 1m 22s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 11s the patch passed
          +1 javadoc 0m 56s the patch passed
          +1 unit 1m 10s hadoop-hdfs-client in the patch passed.
          -1 unit 66m 43s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          100m 38s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
          Timed out junit tests org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-11708
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871485/HDFS-11708-07.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c434a689bccc 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 66c6fd8
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19789/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19789/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19789/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19789/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 8s Maven dependency ordering for branch +1 mvninstall 13m 52s trunk passed +1 compile 1m 31s trunk passed +1 checkstyle 0m 44s trunk passed +1 mvnsite 1m 31s trunk passed +1 findbugs 3m 0s trunk passed +1 javadoc 1m 1s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 20s the patch passed +1 compile 1m 21s the patch passed +1 javac 1m 21s the patch passed -0 checkstyle 0m 40s hadoop-hdfs-project: The patch generated 2 new + 182 unchanged - 0 fixed = 184 total (was 182) +1 mvnsite 1m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 11s the patch passed +1 javadoc 0m 56s the patch passed +1 unit 1m 10s hadoop-hdfs-client in the patch passed. -1 unit 66m 43s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 100m 38s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA Timed out junit tests org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11708 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871485/HDFS-11708-07.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c434a689bccc 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 66c6fd8 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19789/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19789/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19789/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19789/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 5m 59s branch-2.7 passed
          +1 compile 1m 3s branch-2.7 passed with JDK v1.8.0_131
          +1 compile 1m 5s branch-2.7 passed with JDK v1.7.0_131
          +1 checkstyle 0m 24s branch-2.7 passed
          +1 mvnsite 1m 2s branch-2.7 passed
          +1 findbugs 3m 3s branch-2.7 passed
          +1 javadoc 1m 6s branch-2.7 passed with JDK v1.8.0_131
          +1 javadoc 2m 6s branch-2.7 passed with JDK v1.7.0_131
          +1 mvninstall 1m 11s the patch passed
          +1 compile 1m 11s the patch passed with JDK v1.8.0_131
          +1 javac 1m 11s the patch passed
          +1 compile 1m 16s the patch passed with JDK v1.7.0_131
          +1 javac 1m 16s the patch passed
          -0 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 89 unchanged - 0 fixed = 90 total (was 89)
          +1 mvnsite 1m 6s the patch passed
          -1 whitespace 0m 0s The patch has 3479 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 1m 23s The patch 90 line(s) with tabs.
          +1 findbugs 3m 36s the patch passed
          +1 javadoc 0m 57s the patch passed with JDK v1.8.0_131
          +1 javadoc 1m 45s the patch passed with JDK v1.7.0_131
          -1 unit 55m 54s hadoop-hdfs in the patch failed with JDK v1.7.0_131.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          145m 11s



          Reason Tests
          JDK v1.8.0_131 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.hdfs.server.namenode.ha.TestHAAppend
          JDK v1.7.0_131 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.server.namenode.TestFileTruncate



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:67e87c9
          JIRA Issue HDFS-11708
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870781/HDFS-11708-branch-2.7-07.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d8018a3690ed 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7 / db81fbb
          Default Java 1.7.0_131
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19788/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19788/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19788/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19788/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt
          JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19788/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19788/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 5m 59s branch-2.7 passed +1 compile 1m 3s branch-2.7 passed with JDK v1.8.0_131 +1 compile 1m 5s branch-2.7 passed with JDK v1.7.0_131 +1 checkstyle 0m 24s branch-2.7 passed +1 mvnsite 1m 2s branch-2.7 passed +1 findbugs 3m 3s branch-2.7 passed +1 javadoc 1m 6s branch-2.7 passed with JDK v1.8.0_131 +1 javadoc 2m 6s branch-2.7 passed with JDK v1.7.0_131 +1 mvninstall 1m 11s the patch passed +1 compile 1m 11s the patch passed with JDK v1.8.0_131 +1 javac 1m 11s the patch passed +1 compile 1m 16s the patch passed with JDK v1.7.0_131 +1 javac 1m 16s the patch passed -0 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 89 unchanged - 0 fixed = 90 total (was 89) +1 mvnsite 1m 6s the patch passed -1 whitespace 0m 0s The patch has 3479 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 1m 23s The patch 90 line(s) with tabs. +1 findbugs 3m 36s the patch passed +1 javadoc 0m 57s the patch passed with JDK v1.8.0_131 +1 javadoc 1m 45s the patch passed with JDK v1.7.0_131 -1 unit 55m 54s hadoop-hdfs in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 145m 11s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.namenode.ha.TestHAAppend JDK v1.7.0_131 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.namenode.TestFileTruncate Subsystem Report/Notes Docker Image:yetus/hadoop:67e87c9 JIRA Issue HDFS-11708 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870781/HDFS-11708-branch-2.7-07.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d8018a3690ed 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / db81fbb Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19788/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19788/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19788/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19788/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19788/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19788/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 30s Maven dependency ordering for branch
          +1 mvninstall 14m 51s trunk passed
          +1 compile 1m 37s trunk passed
          +1 checkstyle 0m 48s trunk passed
          +1 mvnsite 1m 37s trunk passed
          +1 findbugs 3m 18s trunk passed
          +1 javadoc 1m 7s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 1m 27s the patch passed
          +1 compile 1m 21s the patch passed
          +1 javac 1m 21s the patch passed
          -0 checkstyle 0m 41s hadoop-hdfs-project: The patch generated 2 new + 182 unchanged - 0 fixed = 184 total (was 182)
          +1 mvnsite 1m 23s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 10s the patch passed
          +1 javadoc 0m 56s the patch passed
          +1 unit 1m 10s hadoop-hdfs-client in the patch passed.
          -1 unit 105m 42s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          142m 0s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
          Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-11708
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871485/HDFS-11708-07.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 3f3c23f6cba0 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 66c6fd8
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19790/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19790/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19790/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19790/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 30s Maven dependency ordering for branch +1 mvninstall 14m 51s trunk passed +1 compile 1m 37s trunk passed +1 checkstyle 0m 48s trunk passed +1 mvnsite 1m 37s trunk passed +1 findbugs 3m 18s trunk passed +1 javadoc 1m 7s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 27s the patch passed +1 compile 1m 21s the patch passed +1 javac 1m 21s the patch passed -0 checkstyle 0m 41s hadoop-hdfs-project: The patch generated 2 new + 182 unchanged - 0 fixed = 184 total (was 182) +1 mvnsite 1m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 10s the patch passed +1 javadoc 0m 56s the patch passed +1 unit 1m 10s hadoop-hdfs-client in the patch passed. -1 unit 105m 42s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 142m 0s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11708 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871485/HDFS-11708-07.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3f3c23f6cba0 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 66c6fd8 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19790/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19790/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19790/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19790/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          shv Konstantin Shvachko added a comment -

          Thank you for verifying, Vinayakumar B. Turned out the last two lines were causing my problems, some metadata with "--" and "windows" in it.
          +1 on this jira.

          Show
          shv Konstantin Shvachko added a comment - Thank you for verifying, Vinayakumar B . Turned out the last two lines were causing my problems, some metadata with "--" and "windows" in it. +1 on this jira.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Committed to trunk , branch-2, branch-2.8 and branch-2.7.
          Thanks Brahma Reddy Battula and Konstantin Shvachko for reviews.

          Show
          vinayrpet Vinayakumar B added a comment - Committed to trunk , branch-2, branch-2.8 and branch-2.7. Thanks Brahma Reddy Battula and Konstantin Shvachko for reviews.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11837 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11837/)
          HDFS-11708. Positional read will fail if replicas moved to different DNs (vinayakumarb: rev 70fc6746b326b9a913e8bebca5f5afaf01ab9e11)

          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPread.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11837 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11837/ ) HDFS-11708 . Positional read will fail if replicas moved to different DNs (vinayakumarb: rev 70fc6746b326b9a913e8bebca5f5afaf01ab9e11) (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPread.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java

            People

            • Assignee:
              vinayrpet Vinayakumar B
              Reporter:
              vinayrpet Vinayakumar B
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development