Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6569

OOB message can't be sent to the client when DataNode shuts down for upgrade

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 2.4.0
    • Fix Version/s: 2.6.0
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The socket is closed too early before the OOB message can be sent to client, which causes the write pipeline failure.

      1. HDFS-6569.001.patch
        11 kB
        Brandon Li
      2. HDFS-6569.002.patch
        11 kB
        Brandon Li
      3. HDFS-6569.003.patch
        12 kB
        Brandon Li
      4. test-hdfs-6569.patch
        7 kB
        Brandon Li

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1868 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1868/)
          HDFS-6569. OOB message can't be sent to the client when DataNode shuts down for upgrade. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1618742)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeRollingUpgrade.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1868 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1868/ ) HDFS-6569 . OOB message can't be sent to the client when DataNode shuts down for upgrade. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1618742 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeRollingUpgrade.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1842 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1842/)
          HDFS-6569. OOB message can't be sent to the client when DataNode shuts down for upgrade. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1618742)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeRollingUpgrade.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1842 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1842/ ) HDFS-6569 . OOB message can't be sent to the client when DataNode shuts down for upgrade. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1618742 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeRollingUpgrade.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #651 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/651/)
          HDFS-6569. OOB message can't be sent to the client when DataNode shuts down for upgrade. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1618742)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeRollingUpgrade.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #651 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/651/ ) HDFS-6569 . OOB message can't be sent to the client when DataNode shuts down for upgrade. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1618742 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeRollingUpgrade.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #6086 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6086/)
          HDFS-6569. OOB message can't be sent to the client when DataNode shuts down for upgrade. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1618742)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeRollingUpgrade.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6086 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6086/ ) HDFS-6569 . OOB message can't be sent to the client when DataNode shuts down for upgrade. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1618742 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeRollingUpgrade.java
          Hide
          Brandon Li added a comment -

          Thank you, Kihwal. I've committed the patch.

          Show
          Brandon Li added a comment - Thank you, Kihwal. I've committed the patch.
          Hide
          Kihwal Lee added a comment -

          +1

          Show
          Kihwal Lee added a comment - +1
          Hide
          Brandon Li added a comment -

          The test failure was not introduced by the patch and they passed in my local tests.

          Show
          Brandon Li added a comment - The test failure was not introduced by the patch and they passed in my local tests.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12662194/HDFS-6569.003.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.balancer.TestBalancer
          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover

          The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestRollingUpgradeRollback
          org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
          org.apache.hadoop.hdfs.server.namenode.ha.TestHAMetrics
          org.apache.hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA
          org.apache.hadoop.hdfs.server.namenode.ha.TestHAStateTransitions
          org.apache.hadoop.hdfs.server.namenode.TestValidateConfigurationSettings
          org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup
          org.apache.hadoop.hdfs.TestHDFSServerPorts

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7649//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7649//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12662194/HDFS-6569.003.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.balancer.TestBalancer org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestRollingUpgradeRollback org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints org.apache.hadoop.hdfs.server.namenode.ha.TestHAMetrics org.apache.hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA org.apache.hadoop.hdfs.server.namenode.ha.TestHAStateTransitions org.apache.hadoop.hdfs.server.namenode.TestValidateConfigurationSettings org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup org.apache.hadoop.hdfs.TestHDFSServerPorts +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7649//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7649//console This message is automatically generated.
          Hide
          Brandon Li added a comment -

          Thank you, Kihwal, for the review. I've created HDFS-6856 to track the effort of sending OOB ack asynchronously.
          Uploaded a new patch to not send OOB twice.

          Show
          Brandon Li added a comment - Thank you, Kihwal, for the review. I've created HDFS-6856 to track the effort of sending OOB ack asynchronously. Uploaded a new patch to not send OOB twice.
          Hide
          Kihwal Lee added a comment -

          The patch looks good in general. There are a couple of things that can be improved though.

          • Serial transmission of OOB. One bad client may block this and prevent the message from being sent to the rest of "good" clients. Unless a new thread is created (during shutdown!) to send an OOB ack asynchrously, the blocking ack.readFields() call needs to be changed in order to delegate the message transmission to the responder thread. I believe this is beyond the scope of this jira. I suggest filing a new jira for improving this.
          • Shutdown OOB can be sent twice. This does not affect the correctness, but DN log can become a bit messy. We can make it skip the OOB sending on interrupt, if it was already sent. If you want to address this in a separate jira, that is fine, since it is a minor issue.
          Show
          Kihwal Lee added a comment - The patch looks good in general. There are a couple of things that can be improved though. Serial transmission of OOB. One bad client may block this and prevent the message from being sent to the rest of "good" clients. Unless a new thread is created (during shutdown!) to send an OOB ack asynchrously, the blocking ack.readFields() call needs to be changed in order to delegate the message transmission to the responder thread. I believe this is beyond the scope of this jira. I suggest filing a new jira for improving this. Shutdown OOB can be sent twice. This does not affect the correctness, but DN log can become a bit messy. We can make it skip the OOB sending on interrupt, if it was already sent. If you want to address this in a separate jira, that is fine, since it is a minor issue.
          Hide
          Brandon Li added a comment -

          The test failure was not introduced by this patch.

          Show
          Brandon Li added a comment - The test failure was not introduced by this patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12660826/HDFS-6569.002.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7600//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7600//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12660826/HDFS-6569.002.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7600//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7600//console This message is automatically generated.
          Hide
          Brandon Li added a comment -

          Uploaded the patch to fix the unit tests.

          Show
          Brandon Li added a comment - Uploaded the patch to fix the unit tests.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12660761/HDFS-6569.001.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestSafeMode
          org.apache.hadoop.hdfs.TestHFlush
          org.apache.hadoop.hdfs.TestModTime
          org.apache.hadoop.TestGenericRefresh
          org.apache.hadoop.hdfs.server.datanode.TestReadOnlySharedStorage
          org.apache.hadoop.hdfs.server.datanode.TestIncrementalBlockReports
          org.apache.hadoop.hdfs.server.namenode.ha.TestFailoverWithBlockTokensEnabled
          org.apache.hadoop.hdfs.web.TestHttpsFileSystem
          org.apache.hadoop.cli.TestXAttrCLI
          org.apache.hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotRename
          org.apache.hadoop.hdfs.TestReplication
          org.apache.hadoop.hdfs.server.namenode.TestNameNodeAcl
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyIsHot
          org.apache.hadoop.hdfs.TestAbandonBlock
          org.apache.hadoop.hdfs.TestSetTimes
          org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs
          org.apache.hadoop.net.TestNetworkTopology
          org.apache.hadoop.hdfs.TestDFSFinalize
          org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs
          org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery
          org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement
          org.apache.hadoop.hdfs.server.namenode.TestAuditLogs
          org.apache.hadoop.hdfs.server.namenode.TestStorageRestore
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion
          org.apache.hadoop.hdfs.TestFSOutputSummer
          org.apache.hadoop.hdfs.server.blockmanagement.TestNodeCount
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          org.apache.hadoop.hdfs.TestAppendDifferentChecksum
          org.apache.hadoop.hdfs.server.namenode.TestEditLogRace
          org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart
          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          org.apache.hadoop.hdfs.server.namenode.TestNameNodeResourceChecker
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration
          org.apache.hadoop.hdfs.TestHDFSFileSystemContract
          org.apache.hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport
          org.apache.hadoop.hdfs.TestDFSShell
          org.apache.hadoop.hdfs.TestMissingBlocksAlert
          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
          org.apache.hadoop.hdfs.TestFileCreationClient
          org.apache.hadoop.hdfs.TestClientReportBadBlock
          org.apache.hadoop.fs.contract.hdfs.TestHDFSContractSeek
          org.apache.hadoop.fs.TestEnhancedByteBufferAccess
          org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite
          org.apache.hadoop.hdfs.web.TestWebHDFS
          org.apache.hadoop.hdfs.TestSmallBlock
          org.apache.hadoop.hdfs.server.namenode.TestStartup
          org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus
          org.apache.hadoop.fs.TestFcHdfsCreateMkdir
          org.apache.hadoop.hdfs.server.namenode.TestAuditLogger
          org.apache.hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
          org.apache.hadoop.hdfs.TestReadWhileWriting
          org.apache.hadoop.hdfs.web.TestWebHdfsTokens
          org.apache.hadoop.hdfs.server.namenode.TestMetadataVersionOutput
          org.apache.hadoop.hdfs.server.datanode.TestTransferRbw
          org.apache.hadoop.hdfs.TestFileCreation
          org.apache.hadoop.hdfs.TestRead
          org.apache.hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN
          org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol
          org.apache.hadoop.fs.TestFcHdfsPermission
          org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery
          org.apache.hadoop.hdfs.server.namenode.TestFSEditLogLoader
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics
          org.apache.hadoop.hdfs.TestWriteConfigurationToDFS
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
          org.apache.hadoop.hdfs.server.namenode.TestFileContextAcl
          org.apache.hadoop.hdfs.TestParallelShortCircuitReadNoChecksum
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSetQuotaWithSnapshot
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotStatsMXBean
          org.apache.hadoop.fs.shell.TestHdfsTextCommand
          org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl
          org.apache.hadoop.hdfs.server.namenode.ha.TestHAStateTransitions
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestDisallowModifyROSnapshot
          org.apache.hadoop.hdfs.TestRenameWhileOpen
          org.apache.hadoop.hdfs.TestDFSClientRetries
          org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks
          org.apache.hadoop.hdfs.server.namenode.TestAddBlockRetry
          org.apache.hadoop.hdfs.server.namenode.TestNameNodeXAttr
          org.apache.hadoop.hdfs.TestSnapshotCommands
          org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA
          org.apache.hadoop.hdfs.TestFileAppend
          org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode
          org.apache.hadoop.hdfs.server.namenode.ha.TestHAAppend
          org.apache.hadoop.hdfs.TestLargeBlock
          org.apache.hadoop.hdfs.TestFileCorruption
          org.apache.hadoop.hdfs.TestQuota
          org.apache.hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics
          org.apache.hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate
          org.apache.hadoop.cli.TestHDFSCLI
          org.apache.hadoop.hdfs.TestDFSShellGenericOptions
          org.apache.hadoop.hdfs.TestHDFSServerPorts
          org.apache.hadoop.hdfs.server.namenode.TestXAttrConfigFlag
          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
          org.apache.hadoop.fs.viewfs.TestViewFsWithAcls
          org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes
          org.apache.hadoop.fs.viewfs.TestViewFileSystemWithAcls
          org.apache.hadoop.hdfs.TestLocalDFS
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshot
          org.apache.hadoop.fs.TestSymlinkHdfsFileSystem
          org.apache.hadoop.hdfs.TestLease
          org.apache.hadoop.hdfs.server.datanode.TestStorageReport
          org.apache.hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
          org.apache.hadoop.hdfs.server.namenode.TestFSNamesystemMBean
          org.apache.hadoop.hdfs.TestIsMethodSupported
          org.apache.hadoop.hdfs.TestWriteRead
          org.apache.hadoop.hdfs.server.blockmanagement.TestPendingReplication
          org.apache.hadoop.hdfs.server.datanode.TestCachingStrategy
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
          org.apache.hadoop.hdfs.TestFileAppend2
          org.apache.hadoop.hdfs.server.namenode.TestCheckpoint
          org.apache.hadoop.fs.TestUrlStreamHandler
          org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencing
          org.apache.hadoop.hdfs.TestSetrepIncreasing
          org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
          org.apache.hadoop.hdfs.TestDatanodeDeath
          org.apache.hadoop.hdfs.web.TestWebHdfsWithAuthenticationFilter
          org.apache.hadoop.security.TestRefreshUserMappings
          org.apache.hadoop.hdfs.server.namenode.metrics.TestNNMetricFilesInGetListingOps
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
          org.apache.hadoop.hdfs.TestBlockReaderFactory
          org.apache.hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens
          org.apache.hadoop.fs.viewfs.TestViewFileSystemHdfs
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeMXBean
          org.apache.hadoop.hdfs.server.namenode.TestHostsFiles
          org.apache.hadoop.hdfs.server.namenode.TestBlockUnderConstruction
          org.apache.hadoop.hdfs.server.namenode.TestAddBlock
          org.apache.hadoop.fs.contract.hdfs.TestHDFSContractOpen
          org.apache.hadoop.hdfs.TestBlockReaderLocalLegacy
          org.apache.hadoop.hdfs.server.namenode.TestFSImage
          org.apache.hadoop.fs.viewfs.TestViewFsAtHdfsRoot
          org.apache.hadoop.hdfs.TestClose
          org.apache.hadoop.hdfs.TestFetchImage
          org.apache.hadoop.fs.contract.hdfs.TestHDFSContractDelete
          org.apache.hadoop.hdfs.TestDataTransferKeepalive
          org.apache.hadoop.hdfs.TestDatanodeLayoutUpgrade
          org.apache.hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing
          org.apache.hadoop.hdfs.TestPersistBlocks
          org.apache.hadoop.fs.viewfs.TestViewFileSystemWithXAttrs
          org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica
          org.apache.hadoop.hdfs.server.namenode.TestBackupNode
          org.apache.hadoop.hdfs.server.datanode.TestDiskError
          org.apache.hadoop.hdfs.TestBlockMissingException
          org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
          org.apache.hadoop.hdfs.TestFileAppend4
          org.apache.hadoop.TestRefreshCallQueue
          org.apache.hadoop.hdfs.TestHDFSTrash
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeExit
          org.apache.hadoop.hdfs.TestPread
          org.apache.hadoop.hdfs.TestFileCreationEmpty
          org.apache.hadoop.hdfs.TestFileCreationDelete
          org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner
          org.apache.hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
          org.apache.hadoop.hdfs.TestListFilesInDFS
          org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure
          org.apache.hadoop.hdfs.TestDatanodeConfig
          org.apache.hadoop.security.TestPermissionSymlinks
          org.apache.hadoop.fs.contract.hdfs.TestHDFSContractMkdir
          org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
          org.apache.hadoop.hdfs.server.balancer.TestBalancer
          org.apache.hadoop.hdfs.server.namenode.TestHDFSConcat
          org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
          org.apache.hadoop.hdfs.server.datanode.TestIncrementalBrVariations
          org.apache.hadoop.hdfs.TestDFSUpgradeFromImage
          org.apache.hadoop.hdfs.server.namenode.TestNamenodeRetryCache
          org.apache.hadoop.hdfs.TestDisableConnCache
          org.apache.hadoop.hdfs.TestParallelRead
          org.apache.hadoop.hdfs.server.namenode.TestAclConfigFlag
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
          org.apache.hadoop.hdfs.TestPipelines
          org.apache.hadoop.hdfs.TestCrcCorruption
          org.apache.hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
          org.apache.hadoop.hdfs.TestDFSClientFailover
          org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner
          org.apache.hadoop.hdfs.server.namenode.TestNameNodeMXBean
          org.apache.hadoop.hdfs.security.token.block.TestBlockToken
          org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
          org.apache.hadoop.fs.permission.TestStickyBit
          org.apache.hadoop.hdfs.server.namenode.TestFSImageWithSnapshot
          org.apache.hadoop.hdfs.TestClientBlockVerification
          org.apache.hadoop.hdfs.TestDFSClientExcludedNodes
          org.apache.hadoop.hdfs.TestMultiThreadedHflush
          org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached
          org.apache.hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot
          org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
          org.apache.hadoop.fs.viewfs.TestViewFsHdfs
          org.apache.hadoop.hdfs.server.namenode.TestFSDirectory
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot
          org.apache.hadoop.hdfs.TestFileLengthOnClusterRestart
          org.apache.hadoop.hdfs.TestMiniDFSCluster
          org.apache.hadoop.hdfs.TestFileConcurrentReader
          org.apache.hadoop.cli.TestAclCLI
          org.apache.hadoop.hdfs.TestDFSOutputStream
          org.apache.hadoop.fs.contract.hdfs.TestHDFSContractConcat
          org.apache.hadoop.fs.contract.hdfs.TestHDFSContractAppend
          org.apache.hadoop.hdfs.server.namenode.TestFSImageWithAcl
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters
          org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
          org.apache.hadoop.hdfs.TestDFSMkdirs
          org.apache.hadoop.hdfs.TestDecommission
          org.apache.hadoop.fs.contract.hdfs.TestHDFSContractRootDirectory
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots
          org.apache.hadoop.hdfs.TestListFilesInFileContext
          org.apache.hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
          org.apache.hadoop.hdfs.TestRollingUpgrade
          org.apache.hadoop.hdfs.TestGetBlocks
          org.apache.hadoop.hdfs.TestRestartDFS
          org.apache.hadoop.hdfs.TestDFSStartupVersions
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap
          org.apache.hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages
          org.apache.hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade
          org.apache.hadoop.hdfs.TestDFSRollback
          org.apache.hadoop.hdfs.TestDataTransferProtocol
          org.apache.hadoop.hdfs.TestDFSRename
          org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache
          org.apache.hadoop.hdfs.TestRollingUpgradeDowngrade
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          org.apache.hadoop.hdfs.TestDFSPermission
          org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
          org.apache.hadoop.hdfs.server.namenode.TestFileContextXAttr
          org.apache.hadoop.hdfs.TestDatanodeReport
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
          org.apache.hadoop.hdfs.server.namenode.TestSaveNamespace
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing
          org.apache.hadoop.hdfs.TestDFSAddressConfig
          org.apache.hadoop.cli.TestCacheAdminCLI
          org.apache.hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete
          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
          org.apache.hadoop.hdfs.server.namenode.TestDeleteRace
          org.apache.hadoop.hdfs.server.datanode.TestRefreshNamenodes
          org.apache.hadoop.hdfs.security.TestDelegationToken
          org.apache.hadoop.fs.loadGenerator.TestLoadGenerator
          org.apache.hadoop.hdfs.server.namenode.ha.TestHAMetrics
          org.apache.hadoop.hdfs.TestDatanodeBlockScanner
          org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage
          org.apache.hadoop.hdfs.TestFileAppendRestart
          org.apache.hadoop.hdfs.server.namenode.TestSequentialBlockId
          org.apache.hadoop.hdfs.server.namenode.TestMetaSave
          org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives
          org.apache.hadoop.hdfs.TestRollingUpgradeRollback
          org.apache.hadoop.hdfs.TestEncryptedTransfer
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot
          org.apache.hadoop.hdfs.TestDatanodeRegistration
          org.apache.hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling
          org.apache.hadoop.fs.TestFcHdfsSetUMask
          org.apache.hadoop.hdfs.TestFileAppend3
          org.apache.hadoop.hdfs.server.namenode.ha.TestHarFileSystemWithHA
          org.apache.hadoop.hdfs.server.namenode.TestFsck
          org.apache.hadoop.fs.contract.hdfs.TestHDFSContractRename
          org.apache.hadoop.hdfs.TestBalancerBandwidth
          org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
          org.apache.hadoop.hdfs.TestDFSRemove
          org.apache.hadoop.security.TestPermission
          org.apache.hadoop.fs.viewfs.TestViewFsWithXAttrs
          org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
          org.apache.hadoop.hdfs.TestDistributedFileSystem
          org.apache.hadoop.fs.TestResolveHdfsSymlink
          org.apache.hadoop.hdfs.TestSetrepDecreasing
          org.apache.hadoop.hdfs.TestFSInputChecker
          org.apache.hadoop.hdfs.server.namenode.TestFileLimit
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots
          org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
          org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
          org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode
          org.apache.hadoop.fs.TestGlobPaths
          org.apache.hadoop.hdfs.TestFsShellPermission
          org.apache.hadoop.hdfs.web.TestWebHDFSXAttr
          org.apache.hadoop.hdfs.TestGetFileChecksum
          org.apache.hadoop.hdfs.TestLeaseRecovery
          org.apache.hadoop.hdfs.web.TestWebHDFSAcl
          org.apache.hadoop.fs.contract.hdfs.TestHDFSContractCreate
          org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM
          org.apache.hadoop.hdfs.TestParallelShortCircuitLegacyRead
          org.apache.hadoop.fs.TestHDFSFileContextMainOperations
          org.apache.hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeInitStorage
          org.apache.hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
          org.apache.hadoop.hdfs.TestFileStatus
          org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool
          org.apache.hadoop.hdfs.server.namenode.ha.TestQuotasWithHA
          org.apache.hadoop.hdfs.server.namenode.TestFSImageWithXAttr
          org.apache.hadoop.hdfs.TestLeaseRecovery2
          org.apache.hadoop.hdfs.server.namenode.TestINodeFile
          org.apache.hadoop.hdfs.TestBlockReaderLocal
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot
          org.apache.hadoop.hdfs.TestDFSUpgrade
          org.apache.hadoop.hdfs.web.TestWebHDFSForHA
          org.apache.hadoop.hdfs.server.namenode.TestNameNodeRpcServer
          org.apache.hadoop.hdfs.server.datanode.TestHSync
          org.apache.hadoop.hdfs.TestParallelUnixDomainRead
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication
          org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots
          org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache
          org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
          org.apache.hadoop.hdfs.TestParallelShortCircuitRead
          org.apache.hadoop.fs.TestSymlinkHdfsFileContext
          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
          org.apache.hadoop.hdfs.TestSeekBug
          org.apache.hadoop.hdfs.server.namenode.TestAllowFormat
          org.apache.hadoop.hdfs.server.namenode.TestCreateEditsLog

          The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.tools.TestJMXGet

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7598//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7598//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12660761/HDFS-6569.001.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestSafeMode org.apache.hadoop.hdfs.TestHFlush org.apache.hadoop.hdfs.TestModTime org.apache.hadoop.TestGenericRefresh org.apache.hadoop.hdfs.server.datanode.TestReadOnlySharedStorage org.apache.hadoop.hdfs.server.datanode.TestIncrementalBlockReports org.apache.hadoop.hdfs.server.namenode.ha.TestFailoverWithBlockTokensEnabled org.apache.hadoop.hdfs.web.TestHttpsFileSystem org.apache.hadoop.cli.TestXAttrCLI org.apache.hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotRename org.apache.hadoop.hdfs.TestReplication org.apache.hadoop.hdfs.server.namenode.TestNameNodeAcl org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyIsHot org.apache.hadoop.hdfs.TestAbandonBlock org.apache.hadoop.hdfs.TestSetTimes org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs org.apache.hadoop.net.TestNetworkTopology org.apache.hadoop.hdfs.TestDFSFinalize org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement org.apache.hadoop.hdfs.server.namenode.TestAuditLogs org.apache.hadoop.hdfs.server.namenode.TestStorageRestore org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion org.apache.hadoop.hdfs.TestFSOutputSummer org.apache.hadoop.hdfs.server.blockmanagement.TestNodeCount org.apache.hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots org.apache.hadoop.hdfs.TestAppendDifferentChecksum org.apache.hadoop.hdfs.server.namenode.TestEditLogRace org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.server.namenode.TestNameNodeResourceChecker org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration org.apache.hadoop.hdfs.TestHDFSFileSystemContract org.apache.hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport org.apache.hadoop.hdfs.TestDFSShell org.apache.hadoop.hdfs.TestMissingBlocksAlert org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes org.apache.hadoop.hdfs.TestFileCreationClient org.apache.hadoop.hdfs.TestClientReportBadBlock org.apache.hadoop.fs.contract.hdfs.TestHDFSContractSeek org.apache.hadoop.fs.TestEnhancedByteBufferAccess org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite org.apache.hadoop.hdfs.web.TestWebHDFS org.apache.hadoop.hdfs.TestSmallBlock org.apache.hadoop.hdfs.server.namenode.TestStartup org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus org.apache.hadoop.fs.TestFcHdfsCreateMkdir org.apache.hadoop.hdfs.server.namenode.TestAuditLogger org.apache.hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork org.apache.hadoop.hdfs.TestReadWhileWriting org.apache.hadoop.hdfs.web.TestWebHdfsTokens org.apache.hadoop.hdfs.server.namenode.TestMetadataVersionOutput org.apache.hadoop.hdfs.server.datanode.TestTransferRbw org.apache.hadoop.hdfs.TestFileCreation org.apache.hadoop.hdfs.TestRead org.apache.hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol org.apache.hadoop.fs.TestFcHdfsPermission org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery org.apache.hadoop.hdfs.server.namenode.TestFSEditLogLoader org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics org.apache.hadoop.hdfs.TestWriteConfigurationToDFS org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport org.apache.hadoop.hdfs.server.namenode.TestFileContextAcl org.apache.hadoop.hdfs.TestParallelShortCircuitReadNoChecksum org.apache.hadoop.hdfs.server.namenode.snapshot.TestSetQuotaWithSnapshot org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotStatsMXBean org.apache.hadoop.fs.shell.TestHdfsTextCommand org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl org.apache.hadoop.hdfs.server.namenode.ha.TestHAStateTransitions org.apache.hadoop.hdfs.server.namenode.snapshot.TestDisallowModifyROSnapshot org.apache.hadoop.hdfs.TestRenameWhileOpen org.apache.hadoop.hdfs.TestDFSClientRetries org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks org.apache.hadoop.hdfs.server.namenode.TestAddBlockRetry org.apache.hadoop.hdfs.server.namenode.TestNameNodeXAttr org.apache.hadoop.hdfs.TestSnapshotCommands org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA org.apache.hadoop.hdfs.TestFileAppend org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode org.apache.hadoop.hdfs.server.namenode.ha.TestHAAppend org.apache.hadoop.hdfs.TestLargeBlock org.apache.hadoop.hdfs.TestFileCorruption org.apache.hadoop.hdfs.TestQuota org.apache.hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics org.apache.hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.TestDFSShellGenericOptions org.apache.hadoop.hdfs.TestHDFSServerPorts org.apache.hadoop.hdfs.server.namenode.TestXAttrConfigFlag org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup org.apache.hadoop.fs.viewfs.TestViewFsWithAcls org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes org.apache.hadoop.fs.viewfs.TestViewFileSystemWithAcls org.apache.hadoop.hdfs.TestLocalDFS org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshot org.apache.hadoop.fs.TestSymlinkHdfsFileSystem org.apache.hadoop.hdfs.TestLease org.apache.hadoop.hdfs.server.datanode.TestStorageReport org.apache.hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks org.apache.hadoop.hdfs.server.namenode.TestFSNamesystemMBean org.apache.hadoop.hdfs.TestIsMethodSupported org.apache.hadoop.hdfs.TestWriteRead org.apache.hadoop.hdfs.server.blockmanagement.TestPendingReplication org.apache.hadoop.hdfs.server.datanode.TestCachingStrategy org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.server.namenode.TestCheckpoint org.apache.hadoop.fs.TestUrlStreamHandler org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencing org.apache.hadoop.hdfs.TestSetrepIncreasing org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics org.apache.hadoop.hdfs.TestDatanodeDeath org.apache.hadoop.hdfs.web.TestWebHdfsWithAuthenticationFilter org.apache.hadoop.security.TestRefreshUserMappings org.apache.hadoop.hdfs.server.namenode.metrics.TestNNMetricFilesInGetListingOps org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot org.apache.hadoop.hdfs.TestBlockReaderFactory org.apache.hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens org.apache.hadoop.fs.viewfs.TestViewFileSystemHdfs org.apache.hadoop.hdfs.server.datanode.TestDataNodeMXBean org.apache.hadoop.hdfs.server.namenode.TestHostsFiles org.apache.hadoop.hdfs.server.namenode.TestBlockUnderConstruction org.apache.hadoop.hdfs.server.namenode.TestAddBlock org.apache.hadoop.fs.contract.hdfs.TestHDFSContractOpen org.apache.hadoop.hdfs.TestBlockReaderLocalLegacy org.apache.hadoop.hdfs.server.namenode.TestFSImage org.apache.hadoop.fs.viewfs.TestViewFsAtHdfsRoot org.apache.hadoop.hdfs.TestClose org.apache.hadoop.hdfs.TestFetchImage org.apache.hadoop.fs.contract.hdfs.TestHDFSContractDelete org.apache.hadoop.hdfs.TestDataTransferKeepalive org.apache.hadoop.hdfs.TestDatanodeLayoutUpgrade org.apache.hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing org.apache.hadoop.hdfs.TestPersistBlocks org.apache.hadoop.fs.viewfs.TestViewFileSystemWithXAttrs org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.server.datanode.TestDiskError org.apache.hadoop.hdfs.TestBlockMissingException org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks org.apache.hadoop.hdfs.TestFileAppend4 org.apache.hadoop.TestRefreshCallQueue org.apache.hadoop.hdfs.TestHDFSTrash org.apache.hadoop.hdfs.server.datanode.TestDataNodeExit org.apache.hadoop.hdfs.TestPread org.apache.hadoop.hdfs.TestFileCreationEmpty org.apache.hadoop.hdfs.TestFileCreationDelete org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner org.apache.hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage org.apache.hadoop.hdfs.TestListFilesInDFS org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure org.apache.hadoop.hdfs.TestDatanodeConfig org.apache.hadoop.security.TestPermissionSymlinks org.apache.hadoop.fs.contract.hdfs.TestHDFSContractMkdir org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead org.apache.hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer org.apache.hadoop.hdfs.server.balancer.TestBalancer org.apache.hadoop.hdfs.server.namenode.TestHDFSConcat org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.server.datanode.TestIncrementalBrVariations org.apache.hadoop.hdfs.TestDFSUpgradeFromImage org.apache.hadoop.hdfs.server.namenode.TestNamenodeRetryCache org.apache.hadoop.hdfs.TestDisableConnCache org.apache.hadoop.hdfs.TestParallelRead org.apache.hadoop.hdfs.server.namenode.TestAclConfigFlag org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS org.apache.hadoop.hdfs.TestPipelines org.apache.hadoop.hdfs.TestCrcCorruption org.apache.hadoop.hdfs.server.namenode.TestSecurityTokenEditLog org.apache.hadoop.hdfs.TestDFSClientFailover org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner org.apache.hadoop.hdfs.server.namenode.TestNameNodeMXBean org.apache.hadoop.hdfs.security.token.block.TestBlockToken org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication org.apache.hadoop.fs.permission.TestStickyBit org.apache.hadoop.hdfs.server.namenode.TestFSImageWithSnapshot org.apache.hadoop.hdfs.TestClientBlockVerification org.apache.hadoop.hdfs.TestDFSClientExcludedNodes org.apache.hadoop.hdfs.TestMultiThreadedHflush org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached org.apache.hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks org.apache.hadoop.fs.viewfs.TestViewFsHdfs org.apache.hadoop.hdfs.server.namenode.TestFSDirectory org.apache.hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot org.apache.hadoop.hdfs.TestFileLengthOnClusterRestart org.apache.hadoop.hdfs.TestMiniDFSCluster org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.cli.TestAclCLI org.apache.hadoop.hdfs.TestDFSOutputStream org.apache.hadoop.fs.contract.hdfs.TestHDFSContractConcat org.apache.hadoop.fs.contract.hdfs.TestHDFSContractAppend org.apache.hadoop.hdfs.server.namenode.TestFSImageWithAcl org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport org.apache.hadoop.hdfs.TestDFSMkdirs org.apache.hadoop.hdfs.TestDecommission org.apache.hadoop.fs.contract.hdfs.TestHDFSContractRootDirectory org.apache.hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots org.apache.hadoop.hdfs.TestListFilesInFileContext org.apache.hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer org.apache.hadoop.hdfs.TestRollingUpgrade org.apache.hadoop.hdfs.TestGetBlocks org.apache.hadoop.hdfs.TestRestartDFS org.apache.hadoop.hdfs.TestDFSStartupVersions org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap org.apache.hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages org.apache.hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade org.apache.hadoop.hdfs.TestDFSRollback org.apache.hadoop.hdfs.TestDataTransferProtocol org.apache.hadoop.hdfs.TestDFSRename org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache org.apache.hadoop.hdfs.TestRollingUpgradeDowngrade org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.hdfs.TestDFSPermission org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA org.apache.hadoop.hdfs.server.namenode.TestFileContextXAttr org.apache.hadoop.hdfs.TestDatanodeReport org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength org.apache.hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade org.apache.hadoop.hdfs.server.namenode.TestSaveNamespace org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing org.apache.hadoop.hdfs.TestDFSAddressConfig org.apache.hadoop.cli.TestCacheAdminCLI org.apache.hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer org.apache.hadoop.hdfs.server.namenode.TestDeleteRace org.apache.hadoop.hdfs.server.datanode.TestRefreshNamenodes org.apache.hadoop.hdfs.security.TestDelegationToken org.apache.hadoop.fs.loadGenerator.TestLoadGenerator org.apache.hadoop.hdfs.server.namenode.ha.TestHAMetrics org.apache.hadoop.hdfs.TestDatanodeBlockScanner org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage org.apache.hadoop.hdfs.TestFileAppendRestart org.apache.hadoop.hdfs.server.namenode.TestSequentialBlockId org.apache.hadoop.hdfs.server.namenode.TestMetaSave org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives org.apache.hadoop.hdfs.TestRollingUpgradeRollback org.apache.hadoop.hdfs.TestEncryptedTransfer org.apache.hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot org.apache.hadoop.hdfs.TestDatanodeRegistration org.apache.hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling org.apache.hadoop.fs.TestFcHdfsSetUMask org.apache.hadoop.hdfs.TestFileAppend3 org.apache.hadoop.hdfs.server.namenode.ha.TestHarFileSystemWithHA org.apache.hadoop.hdfs.server.namenode.TestFsck org.apache.hadoop.fs.contract.hdfs.TestHDFSContractRename org.apache.hadoop.hdfs.TestBalancerBandwidth org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock org.apache.hadoop.hdfs.TestDFSRemove org.apache.hadoop.security.TestPermission org.apache.hadoop.fs.viewfs.TestViewFsWithXAttrs org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.fs.TestResolveHdfsSymlink org.apache.hadoop.hdfs.TestSetrepDecreasing org.apache.hadoop.hdfs.TestFSInputChecker org.apache.hadoop.hdfs.server.namenode.TestFileLimit org.apache.hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode org.apache.hadoop.fs.TestGlobPaths org.apache.hadoop.hdfs.TestFsShellPermission org.apache.hadoop.hdfs.web.TestWebHDFSXAttr org.apache.hadoop.hdfs.TestGetFileChecksum org.apache.hadoop.hdfs.TestLeaseRecovery org.apache.hadoop.hdfs.web.TestWebHDFSAcl org.apache.hadoop.fs.contract.hdfs.TestHDFSContractCreate org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM org.apache.hadoop.hdfs.TestParallelShortCircuitLegacyRead org.apache.hadoop.fs.TestHDFSFileContextMainOperations org.apache.hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics org.apache.hadoop.hdfs.server.datanode.TestDataNodeInitStorage org.apache.hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality org.apache.hadoop.hdfs.TestFileStatus org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool org.apache.hadoop.hdfs.server.namenode.ha.TestQuotasWithHA org.apache.hadoop.hdfs.server.namenode.TestFSImageWithXAttr org.apache.hadoop.hdfs.TestLeaseRecovery2 org.apache.hadoop.hdfs.server.namenode.TestINodeFile org.apache.hadoop.hdfs.TestBlockReaderLocal org.apache.hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot org.apache.hadoop.hdfs.TestDFSUpgrade org.apache.hadoop.hdfs.web.TestWebHDFSForHA org.apache.hadoop.hdfs.server.namenode.TestNameNodeRpcServer org.apache.hadoop.hdfs.server.datanode.TestHSync org.apache.hadoop.hdfs.TestParallelUnixDomainRead org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser org.apache.hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy org.apache.hadoop.hdfs.TestParallelShortCircuitRead org.apache.hadoop.fs.TestSymlinkHdfsFileContext org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes org.apache.hadoop.hdfs.TestSeekBug org.apache.hadoop.hdfs.server.namenode.TestAllowFormat org.apache.hadoop.hdfs.server.namenode.TestCreateEditsLog The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.tools.TestJMXGet +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7598//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7598//console This message is automatically generated.
          Hide
          Kihwal Lee added a comment -

          Sorry for the delay. I looked at the patch and the approach is similar to what I was thinking. I will work on it today.

          Show
          Kihwal Lee added a comment - Sorry for the delay. I looked at the patch and the approach is similar to what I was thinking. I will work on it today.
          Hide
          Brandon Li added a comment -

          Uploaded some test code changes I used last week when debugging the problem. The test passed with it.

          Show
          Brandon Li added a comment - Uploaded some test code changes I used last week when debugging the problem. The test passed with it.
          Hide
          Kihwal Lee added a comment -

          TestClientProtocolForPipelineRecovery#testPipelineRecoveryOnOOB is supposed to test the case, but in this test case the thread will be in epoll_wait(), not read(). I guess that's the difference.

          One possible fix is to send OOB message before interrupting DataXceiver threads.

          I explored this approach before, I will see what I can do.

          Show
          Kihwal Lee added a comment - TestClientProtocolForPipelineRecovery#testPipelineRecoveryOnOOB is supposed to test the case, but in this test case the thread will be in epoll_wait(), not read(). I guess that's the difference. One possible fix is to send OOB message before interrupting DataXceiver threads. I explored this approach before, I will see what I can do.
          Hide
          Brandon Li added a comment -

          The current code looks good logically and it tries not closing streams before the OOB is sent.

          I think problem is triggered by the NIO implementation. When DataNode is shutdown for restart, it interrupts all the DataXceiver threads. The NIO channel in NioInetPeer are bound to these threads doing the block receiving. If these threads are interrupted, the stream / channel is closed due to IO safety issues.

          So once the DataXceiver thread is interrupted, rarely the OOB can be sent before NIO channel is closed automatically.
          One possible fix is to send OOB message before interrupting DataXceiver threads.
          Thoughts?

          Show
          Brandon Li added a comment - The current code looks good logically and it tries not closing streams before the OOB is sent. I think problem is triggered by the NIO implementation. When DataNode is shutdown for restart, it interrupts all the DataXceiver threads. The NIO channel in NioInetPeer are bound to these threads doing the block receiving. If these threads are interrupted, the stream / channel is closed due to IO safety issues. So once the DataXceiver thread is interrupted, rarely the OOB can be sent before NIO channel is closed automatically. One possible fix is to send OOB message before interrupting DataXceiver threads. Thoughts?
          Hide
          Brandon Li added a comment -

          Here is the steps to reproduce it:
          1. upload 1GB file using "hadoop dfs -put"
          2. during the file loading, do a "hadoop dfsadmin -shutdownDatanode host:port upgrade".

          This will cause the write failure.

          Show
          Brandon Li added a comment - Here is the steps to reproduce it: 1. upload 1GB file using "hadoop dfs -put" 2. during the file loading, do a "hadoop dfsadmin -shutdownDatanode host:port upgrade". This will cause the write failure.
          Hide
          Brandon Li added a comment -

          Yes. It's a single node cluster with replication factor as 1.

          Show
          Brandon Li added a comment - Yes. It's a single node cluster with replication factor as 1.
          Hide
          Kihwal Lee added a comment -

          It is unclear who closed the connection. During the development, I fixed one multiple closure bug and exceptions caused by extra ones in datanode. There may be other cases remaining. Is it from a write using one replica?

          Show
          Kihwal Lee added a comment - It is unclear who closed the connection. During the development, I fixed one multiple closure bug and exceptions caused by extra ones in datanode. There may be other cases remaining. Is it from a write using one replica?
          Hide
          Brandon Li added a comment -

          The client stack trace:

          14/06/19 11:26:23 WARN hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block BP-1063118919-10.11.1.167-1403201799003:blk_1073741828_1004
          java.io.EOFException: Premature EOF: no length prefix available
          	at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1997)
          	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
          	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:793)
          14/06/19 11:26:23 WARN hdfs.DFSClient: DataStreamer Exception
          java.io.IOException: Broken pipe
          	at sun.nio.ch.FileDispatcher.write0(Native Method)
          	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
          	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69)
          	at sun.nio.ch.IOUtil.write(IOUtil.java:40)
          	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:336)
          	at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
          	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
          	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
          	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
          	at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
          	at java.io.DataOutputStream.write(DataOutputStream.java:90)
          	at org.apache.hadoop.hdfs.DFSOutputStream$Packet.writeTo(DFSOutputStream.java:271)
          	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:574)
          put: All datanodes 127.0.0.1:50010 are bad. Aborting...
          

          The DataNode error:

          2014-06-19 11:26:23,668 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Shutting down for restart (BP-1063118919-10.11.1.167-1403201799003:blk_1073741828_1004).
          2014-06-19 11:26:23,669 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Sending an out of band ack of type OOB_RESTART
          2014-06-19 11:26:23,669 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Error sending OOB Ack.
          java.io.IOException: The stream is closed
                  at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:118)
                  at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
                  at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
                  at java.io.DataOutputStream.flush(DataOutputStream.java:106)
                  at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1339)
                  at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendOOBResponse(BlockReceiver.java:1041)
                  at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:802)
                  at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
                  at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
                  at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
                  at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
                  at java.lang.Thread.run(Thread.java:695)
          
          
          Show
          Brandon Li added a comment - The client stack trace: 14/06/19 11:26:23 WARN hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block BP-1063118919-10.11.1.167-1403201799003:blk_1073741828_1004 java.io.EOFException: Premature EOF: no length prefix available at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1997) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:793) 14/06/19 11:26:23 WARN hdfs.DFSClient: DataStreamer Exception java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:336) at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159) at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117) at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.apache.hadoop.hdfs.DFSOutputStream$Packet.writeTo(DFSOutputStream.java:271) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:574) put: All datanodes 127.0.0.1:50010 are bad. Aborting... The DataNode error: 2014-06-19 11:26:23,668 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Shutting down for restart (BP-1063118919-10.11.1.167-1403201799003:blk_1073741828_1004). 2014-06-19 11:26:23,669 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Sending an out of band ack of type OOB_RESTART 2014-06-19 11:26:23,669 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Error sending OOB Ack. java.io.IOException: The stream is closed at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:118) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) at java.io.DataOutputStream.flush(DataOutputStream.java:106) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1339) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendOOBResponse(BlockReceiver.java:1041) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:802) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234) at java.lang.Thread.run(Thread.java:695)

            People

            • Assignee:
              Brandon Li
              Reporter:
              Brandon Li
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development