Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6616

bestNode shouldn't always return the first DataNode

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: webhdfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When we are doing distcp between clusters, job failed:
      014-06-30 20:56:28,430 INFO org.apache.hadoop.tools.DistCp: FAIL part-r-00101.avro : java.net.NoRouteToHostException: No route to host
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1491)
      at java.security.AccessController.doPrivileged(Native Method)
      at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1485)
      at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1139)
      at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379)
      at org.apache.hadoop.hdfs.HftpFileSystem.open(HftpFileSystem.java:322)
      at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:427)
      at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:419)
      at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:547)
      at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:314)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
      at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
      at org.apache.hadoop.mapred.Child.main(Child.java:249)

      The root reason is one of the DataNode can't access from outside, but inside cluster, it's health.
      In NamenodeWebHdfsMethods.java:bestNode, it always return the first DataNode, so even after the distcp retries, it still failed.

      1. HDFS-6616.patch
        2 kB
        zhaoyunjiong
      2. HDFS-6616.3.patch
        25 kB
        zhaoyunjiong
      3. HDFS-6616.2.patch
        25 kB
        zhaoyunjiong
      4. HDFS-6616.1.patch
        22 kB
        zhaoyunjiong

        Issue Links

          Activity

          zhaoyunjiong created issue -
          Hide
          zhaoyunjiong added a comment -

          One possible solution is choose DataNode randomly with the cost of ignore the network distance.

          Show
          zhaoyunjiong added a comment - One possible solution is choose DataNode randomly with the cost of ignore the network distance.
          zhaoyunjiong made changes -
          Field Original Value New Value
          Attachment HDFS-6616.patch [ 12653337 ]
          zhaoyunjiong made changes -
          Attachment HDFS-6616.patch [ 12653344 ]
          zhaoyunjiong made changes -
          Attachment HDFS-6616.patch [ 12653337 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue is related to HDFS-5891 [ HDFS-5891 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I think it is not a good idea to do random since it will lose data locality, especially, when both the client and the data are in the same host.

          We probably should support exclude nodes in WebHDFS.

          Show
          Tsz Wo Nicholas Sze added a comment - I think it is not a good idea to do random since it will lose data locality, especially, when both the client and the data are in the same host. We probably should support exclude nodes in WebHDFS.
          Tsz Wo Nicholas Sze made changes -
          Component/s webhdfs [ 12319200 ]
          Hide
          zhaoyunjiong added a comment -

          What happened on our cluster is very rare case.
          Server use HDP2.1 and client use HDP1.3, so I come up this patch.

          Correct me if I'm wrong: when using WebHDFS, I think it will be very rare that both client and the data will be in the same host.
          But I agree with you support exclude nodes in WebHDFS is a better idea.

          Show
          zhaoyunjiong added a comment - What happened on our cluster is very rare case. Server use HDP2.1 and client use HDP1.3, so I come up this patch. Correct me if I'm wrong: when using WebHDFS, I think it will be very rare that both client and the data will be in the same host. But I agree with you support exclude nodes in WebHDFS is a better idea.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > Correct me if I'm wrong: when using WebHDFS, I think it will be very rare that both client and the data will be in the same host.

          Client and data will be collocated when WebHDFS is used in MapReduce/YARN jobs. When will webhdfs:// be used instead of hdfs:// ? DistCp for coping data across clusters running different Hadoop versions.

          Show
          Tsz Wo Nicholas Sze added a comment - > Correct me if I'm wrong: when using WebHDFS, I think it will be very rare that both client and the data will be in the same host. Client and data will be collocated when WebHDFS is used in MapReduce/YARN jobs. When will webhdfs:// be used instead of hdfs:// ? DistCp for coping data across clusters running different Hadoop versions.
          Hide
          zhaoyunjiong added a comment -

          Yes. You are right.
          I never thought user may use WebHDFS as source and target filesystem, and running distcp job on source cluster.
          For our use case, we always run jobs on target cluster and use WebHDFS as source filesystem.

          Show
          zhaoyunjiong added a comment - Yes. You are right. I never thought user may use WebHDFS as source and target filesystem, and running distcp job on source cluster. For our use case, we always run jobs on target cluster and use WebHDFS as source filesystem.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          @ zhaoyunjiong, are you going to post a new patch?

          Show
          Tsz Wo Nicholas Sze added a comment - @ zhaoyunjiong, are you going to post a new patch?
          Hide
          zhaoyunjiong added a comment -

          Update patch to support exclude nodes in WebHDFS.

          Show
          zhaoyunjiong added a comment - Update patch to support exclude nodes in WebHDFS.
          zhaoyunjiong made changes -
          Attachment HDFS-6616.1.patch [ 12656006 ]
          Tsz Wo Nicholas Sze made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12656006/HDFS-6616.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.balancer.TestBalancer
          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7364//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7364//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12656006/HDFS-6616.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.balancer.TestBalancer org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7364//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7364//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Patch looks good. Some comments:

          • Let's set ExcludeDatanodesParam.NAME to "excludedatanodes".
          • We should also change WebHdfsFileSystem to use the exclude datanode feature so that it will retry will different datanodes.
          • Please take a look the test failures. I think they should not be related but why they failed?
          Show
          Tsz Wo Nicholas Sze added a comment - Patch looks good. Some comments: Let's set ExcludeDatanodesParam.NAME to "excludedatanodes". We should also change WebHdfsFileSystem to use the exclude datanode feature so that it will retry will different datanodes. Please take a look the test failures. I think they should not be related but why they failed?
          Hide
          Jing Zhao added a comment -

          We should also change WebHdfsFileSystem to use the exclude datanode feature so that it will retry will different datanodes.

          Yeah, I have the same comment. And the excludedatanodes should only be used by the internal retry logic in webhdfsfilesystem instead of an external API.

          Show
          Jing Zhao added a comment - We should also change WebHdfsFileSystem to use the exclude datanode feature so that it will retry will different datanodes. Yeah, I have the same comment. And the excludedatanodes should only be used by the internal retry logic in webhdfsfilesystem instead of an external API.
          Hide
          zhaoyunjiong added a comment -

          Thanks Tsz Wo Nicholas Sze & Jing Zhao.

          Update patch according to comments: change ExcludeDatanodesParam.NAME to "excludedatanodes" and change WebHdfsFileSystem to use the exclude datanode feature.

          The test failures is not related.

          Show
          zhaoyunjiong added a comment - Thanks Tsz Wo Nicholas Sze & Jing Zhao. Update patch according to comments: change ExcludeDatanodesParam.NAME to "excludedatanodes" and change WebHdfsFileSystem to use the exclude datanode feature. The test failures is not related.
          zhaoyunjiong made changes -
          Attachment HDFS-6616.2.patch [ 12656424 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Thanks for the update. Some comments on WebHdfsFileSystem:

          • Should we catch IOException instead of SocketException? I think the redirect datanode should be excluded for any IOException.

          Minors:

          • TYPO: excludns
          • hostName and port can be combined to a single variable, say "redirectHost".
          • There is a tailing "," in ExcludeDatanodesParam if it is not empty. See if you want to get rid of it.

          Jing Zhao, do you want to take a look?

          Show
          Tsz Wo Nicholas Sze added a comment - Thanks for the update. Some comments on WebHdfsFileSystem: Should we catch IOException instead of SocketException? I think the redirect datanode should be excluded for any IOException. Minors: TYPO: excludns hostName and port can be combined to a single variable, say "redirectHost". There is a tailing "," in ExcludeDatanodesParam if it is not empty. See if you want to get rid of it. Jing Zhao , do you want to take a look?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12656424/HDFS-6616.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
          org.apache.hadoop.hdfs.web.TestWebHdfsTimeouts
          org.apache.hadoop.hdfs.web.TestHttpsFileSystem
          org.apache.hadoop.fs.TestSymlinkHdfsFileContext
          org.apache.hadoop.hdfs.TestDistributedFileSystem
          org.apache.hadoop.hdfs.server.namenode.TestAuditLogs
          org.apache.hadoop.fs.TestSymlinkHdfsFileSystem
          org.apache.hadoop.hdfs.web.TestWebHdfsWithAuthenticationFilter
          org.apache.hadoop.hdfs.web.TestWebHDFSAcl
          org.apache.hadoop.hdfs.TestQuota
          org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser
          org.apache.hadoop.hdfs.web.TestWebHDFS
          org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl
          org.apache.hadoop.hdfs.web.TestWebHDFSForHA
          org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs
          org.apache.hadoop.hdfs.web.TestWebHdfsContentLength
          org.apache.hadoop.hdfs.security.TestDelegationToken
          org.apache.hadoop.hdfs.web.TestWebHDFSXAttr
          org.apache.hadoop.hdfs.web.TestWebHdfsTokens
          org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
          org.apache.hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7378//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7378//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12656424/HDFS-6616.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract org.apache.hadoop.hdfs.web.TestWebHdfsTimeouts org.apache.hadoop.hdfs.web.TestHttpsFileSystem org.apache.hadoop.fs.TestSymlinkHdfsFileContext org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.server.namenode.TestAuditLogs org.apache.hadoop.fs.TestSymlinkHdfsFileSystem org.apache.hadoop.hdfs.web.TestWebHdfsWithAuthenticationFilter org.apache.hadoop.hdfs.web.TestWebHDFSAcl org.apache.hadoop.hdfs.TestQuota org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser org.apache.hadoop.hdfs.web.TestWebHDFS org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl org.apache.hadoop.hdfs.web.TestWebHDFSForHA org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs org.apache.hadoop.hdfs.web.TestWebHdfsContentLength org.apache.hadoop.hdfs.security.TestDelegationToken org.apache.hadoop.hdfs.web.TestWebHDFSXAttr org.apache.hadoop.hdfs.web.TestWebHdfsTokens org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer org.apache.hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7378//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7378//console This message is automatically generated.
          Hide
          zhaoyunjiong added a comment -

          Update patch according to comments and fix test failures.

          Show
          zhaoyunjiong added a comment - Update patch according to comments and fix test failures.
          zhaoyunjiong made changes -
          Attachment HDFS-6616.3.patch [ 12656469 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12656469/HDFS-6616.3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7381//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7381//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12656469/HDFS-6616.3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7381//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7381//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
          Tsz Wo Nicholas Sze made changes -
          Hadoop Flags Reviewed [ 10343 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks Yunjiong!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks Yunjiong!
          Tsz Wo Nicholas Sze made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 2.6.0 [ 12327181 ]
          Resolution Fixed [ 1 ]
          Hide
          Jitendra Nath Pandey added a comment -

          Should it be included in 2.5 as well?

          Show
          Jitendra Nath Pandey added a comment - Should it be included in 2.5 as well?
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #5916 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5916/)
          HDFS-6616. Add exclude-datanodes feature to WebHDFS redirection so that it will not redirect retries to the same datanode. Contributed by zhaoyunjiong (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611750)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/ExcludeDatanodesParam.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/web/resources/TestWebHdfsDataLocality.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #5916 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5916/ ) HDFS-6616 . Add exclude-datanodes feature to WebHDFS redirection so that it will not redirect retries to the same datanode. Contributed by zhaoyunjiong (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611750 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/ExcludeDatanodesParam.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/web/resources/TestWebHdfsDataLocality.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #617 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/617/)
          HDFS-6616. Add exclude-datanodes feature to WebHDFS redirection so that it will not redirect retries to the same datanode. Contributed by zhaoyunjiong (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611750)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/ExcludeDatanodesParam.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/web/resources/TestWebHdfsDataLocality.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #617 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/617/ ) HDFS-6616 . Add exclude-datanodes feature to WebHDFS redirection so that it will not redirect retries to the same datanode. Contributed by zhaoyunjiong (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611750 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/ExcludeDatanodesParam.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/web/resources/TestWebHdfsDataLocality.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1809 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1809/)
          HDFS-6616. Add exclude-datanodes feature to WebHDFS redirection so that it will not redirect retries to the same datanode. Contributed by zhaoyunjiong (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611750)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/ExcludeDatanodesParam.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/web/resources/TestWebHdfsDataLocality.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1809 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1809/ ) HDFS-6616 . Add exclude-datanodes feature to WebHDFS redirection so that it will not redirect retries to the same datanode. Contributed by zhaoyunjiong (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611750 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/ExcludeDatanodesParam.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/web/resources/TestWebHdfsDataLocality.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1836 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1836/)
          HDFS-6616. Add exclude-datanodes feature to WebHDFS redirection so that it will not redirect retries to the same datanode. Contributed by zhaoyunjiong (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611750)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/ExcludeDatanodesParam.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/web/resources/TestWebHdfsDataLocality.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1836 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1836/ ) HDFS-6616 . Add exclude-datanodes feature to WebHDFS redirection so that it will not redirect retries to the same datanode. Contributed by zhaoyunjiong (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611750 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/ExcludeDatanodesParam.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/web/resources/TestWebHdfsDataLocality.java
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              zhaoyunjiong
              Reporter:
              zhaoyunjiong
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development