Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9107

Prevent NN's unrecoverable death spiral after full GC

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:

      Description

      A full GC pause in the NN that exceeds the dead node interval can lead to an infinite cycle of full GCs. The most common situation that precipitates an unrecoverable state is a network issue that temporarily cuts off multiple racks.

      The NN wakes up and falsely starts marking nodes dead. This bloats the replication queues which increases memory pressure. The replications create a flurry of incremental block reports and a glut of over-replicated blocks.

      The "dead" nodes heartbeat within seconds. The NN forces a re-registration which requires a full block report - more memory pressure. The NN now has to invalidate all the over-replicated blocks. The extra blocks are added to invalidation queues, tracked in an excess blocks map, etc - much more memory pressure.

      All the memory pressure can push the NN into another full GC which repeats the entire cycle.

      1. HDFS-9107.patch
        6 kB
        Daryn Sharp
      2. HDFS-9107.patch
        1 kB
        Daryn Sharp

        Activity

        Hide
        esteban Esteban Gutierrez added a comment -

        Perhaps us the JvmPauseMonitor to monitor for a large pause and delay the expiration if above a threshold?

        Show
        esteban Esteban Gutierrez added a comment - Perhaps us the JvmPauseMonitor to monitor for a large pause and delay the expiration if above a threshold?
        Hide
        daryn Daryn Sharp added a comment -

        Not perfect, but simple way to prevent nodes being marked dead. Tried other fancy techniques with an exponential moving average to delay based on excess pause time. Settled on simple.

        Writing a test will require heavy instrumentation of the code. Hopefully it's simple enough that I'll get a free pass.

        Show
        daryn Daryn Sharp added a comment - Not perfect, but simple way to prevent nodes being marked dead. Tried other fancy techniques with an exponential moving average to delay based on excess pause time. Settled on simple. Writing a test will require heavy instrumentation of the code. Hopefully it's simple enough that I'll get a free pass.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 17m 50s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 56s There were no new javac warning messages.
        +1 javadoc 10m 18s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 22s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 29s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 12s Pre-build of native portion
        -1 hdfs tests 42m 47s Tests failed in hadoop-hdfs.
            88m 26s  



        Reason Tests
        Failed unit tests hadoop.hdfs.TestFileStatus
          hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
          hadoop.hdfs.server.namenode.TestINodeFile
          hadoop.fs.contract.hdfs.TestHDFSContractOpen
          hadoop.hdfs.server.datanode.TestFsDatasetCache
          hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart
          hadoop.hdfs.TestFileCreationDelete
          hadoop.hdfs.server.namenode.ha.TestHASafeMode
          hadoop.hdfs.TestEncryptionZonesWithHA
          hadoop.fs.contract.hdfs.TestHDFSContractMkdir
          hadoop.hdfs.server.namenode.TestNameNodeXAttr
          hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
          hadoop.hdfs.shortcircuit.TestShortCircuitCache
          hadoop.hdfs.TestDecommission
          hadoop.hdfs.server.namenode.TestFSEditLogLoader
          hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA
          hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
          hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages
          hadoop.hdfs.server.datanode.TestCachingStrategy
          hadoop.hdfs.server.datanode.TestDirectoryScanner
          hadoop.cli.TestXAttrCLI
          hadoop.hdfs.server.namenode.TestDeleteRace
          hadoop.hdfs.server.namenode.TestParallelImageWrite
          hadoop.hdfs.server.namenode.TestNameNodeRespectsBindHostKeys
          hadoop.hdfs.server.namenode.TestNNStorageRetentionFunctional
          hadoop.hdfs.server.namenode.TestSaveNamespace
          hadoop.hdfs.TestDFSRename
          hadoop.hdfs.util.TestDiff
          hadoop.hdfs.server.datanode.TestDataNodeFSDataSetSink
          hadoop.hdfs.server.namenode.TestFsck
          hadoop.hdfs.server.namenode.ha.TestHarFileSystemWithHA
          hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration
          hadoop.hdfs.server.datanode.TestDeleteBlockPool
          hadoop.hdfs.TestRemoteBlockReader2
          hadoop.hdfs.server.namenode.TestStorageRestore
          hadoop.hdfs.server.namenode.TestFileLimit
          hadoop.hdfs.server.blockmanagement.TestNodeCount
          hadoop.fs.contract.hdfs.TestHDFSContractSetTimes
          hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots
          hadoop.hdfs.qjournal.TestNNWithQJM
          hadoop.hdfs.server.namenode.TestFSPermissionChecker
          hadoop.hdfs.server.namenode.TestSecureNameNode
          hadoop.hdfs.server.namenode.TestFileContextAcl
          hadoop.hdfs.server.datanode.TestDataXceiverLazyPersistHint
          hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
          hadoop.hdfs.TestDataTransferProtocol
          hadoop.fs.viewfs.TestViewFsWithXAttrs
          hadoop.hdfs.server.datanode.TestDiskError
          hadoop.hdfs.TestFsShellPermission
          hadoop.hdfs.server.namenode.TestMalformedURLs
          hadoop.hdfs.TestReadWhileWriting
          hadoop.fs.TestSWebHdfsFileContextMainOperations
          hadoop.hdfs.TestIsMethodSupported
          hadoop.hdfs.TestParallelShortCircuitReadNoChecksum
          hadoop.hdfs.server.blockmanagement.TestAvailableSpaceBlockPlacementPolicy
          hadoop.hdfs.TestFileCreationClient
          hadoop.cli.TestAclCLI
          hadoop.hdfs.TestFSOutputSummer
          hadoop.fs.TestHDFSFileContextMainOperations
          hadoop.hdfs.TestDFSStartupVersions
          hadoop.hdfs.server.namenode.ha.TestFailureOfSharedDir
          hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters
          hadoop.hdfs.TestParallelShortCircuitLegacyRead
          hadoop.hdfs.server.blockmanagement.TestBlockManager
          hadoop.hdfs.server.namenode.ha.TestStateTransitionFailure
          hadoop.hdfs.server.namenode.TestGenericJournalConf
          hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
          hadoop.hdfs.TestFileAppend2
          hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory
          hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup
          hadoop.hdfs.server.datanode.TestSimulatedFSDataset
          hadoop.hdfs.server.datanode.TestTransferRbw
          hadoop.fs.contract.hdfs.TestHDFSContractSeek
          hadoop.hdfs.TestLocalDFS
          hadoop.hdfs.server.mover.TestStorageMover
          hadoop.hdfs.server.datanode.TestDataStorage
          hadoop.hdfs.server.namenode.TestTruncateQuotaUpdate
          hadoop.hdfs.server.namenode.TestMetadataVersionOutput
          hadoop.hdfs.server.namenode.ha.TestHAAppend
          hadoop.hdfs.server.namenode.TestFSImageWithAcl
          hadoop.fs.TestUnbuffer
          hadoop.hdfs.TestDFSPermission
          hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
          hadoop.hdfs.server.namenode.TestAddBlock
          hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles
          hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold
          hadoop.hdfs.server.namenode.TestMetaSave
          hadoop.hdfs.TestSmallBlock
          hadoop.hdfs.tools.TestDFSHAAdmin
          hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
          hadoop.tracing.TestTracing
          hadoop.hdfs.server.datanode.TestBPOfferService
          hadoop.hdfs.server.namenode.TestCommitBlockSynchronization
          hadoop.hdfs.server.datanode.TestDatanodeRegister
          hadoop.hdfs.server.datanode.TestDataDirs
          hadoop.hdfs.server.datanode.TestTriggerBlockReport
          hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter
          hadoop.fs.contract.hdfs.TestHDFSContractConcat
          hadoop.hdfs.TestReplication
          hadoop.hdfs.server.namenode.TestEditLog
          hadoop.security.TestRefreshUserMappings
          hadoop.hdfs.server.namenode.snapshot.TestFileContextSnapshot
          hadoop.hdfs.server.mover.TestMover
          hadoop.hdfs.tools.TestDFSZKFailoverController
          hadoop.hdfs.TestDefaultNameNodePort
          hadoop.hdfs.server.namenode.TestHDFSConcat
          hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot
          hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
          hadoop.hdfs.server.namenode.TestFSImageWithXAttr
          hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs
          hadoop.security.TestPermissionSymlinks
          hadoop.hdfs.TestHFlush
          hadoop.hdfs.server.namenode.TestDecommissioningStatus
          hadoop.fs.TestSymlinkHdfsDisable
          hadoop.hdfs.server.namenode.TestQuotaByStorageType
          hadoop.hdfs.TestFileCreation
          hadoop.hdfs.server.namenode.snapshot.TestDisallowModifyROSnapshot
          hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
          hadoop.hdfs.server.namenode.ha.TestQuotasWithHA
          hadoop.fs.TestSymlinkHdfsFileContext
          hadoop.hdfs.qjournal.TestSecureNNWithQJM
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits
          hadoop.hdfs.server.namenode.ha.TestHAStateTransitions
          hadoop.hdfs.server.namenode.snapshot.TestSetQuotaWithSnapshot
          hadoop.hdfs.server.namenode.metrics.TestNNMetricFilesInGetListingOps
          hadoop.hdfs.TestRollingUpgradeDowngrade
          hadoop.hdfs.server.namenode.ha.TestEditLogsDuringFailover
          hadoop.hdfs.TestBlockReaderFactory
          hadoop.hdfs.TestDFSOutputStream
          hadoop.fs.TestUrlStreamHandler
          hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot
          hadoop.hdfs.TestModTime
          hadoop.hdfs.server.namenode.ha.TestEditLogTailer
          hadoop.fs.viewfs.TestViewFileSystemWithXAttrs
          hadoop.hdfs.TestAppendSnapshotTruncate
          hadoop.hdfs.server.namenode.TestNameNodeRecovery
          hadoop.hdfs.TestRestartDFS
          hadoop.hdfs.TestDistributedFileSystem
          hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
          hadoop.hdfs.TestInjectionForSimulatedStorage
          hadoop.hdfs.TestMiniDFSCluster
          hadoop.hdfs.server.datanode.TestFsDatasetCacheRevocation
          hadoop.hdfs.server.namenode.snapshot.TestSnapshot
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          hadoop.hdfs.server.namenode.TestProtectedDirectories
          hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot
          hadoop.hdfs.server.namenode.TestNameNodeMetricsLogger
          hadoop.hdfs.server.balancer.TestBalancer
          hadoop.hdfs.server.namenode.TestAuditLogs
          hadoop.hdfs.server.namenode.TestFileTruncate
          hadoop.fs.TestWebHdfsFileContextMainOperations
          hadoop.hdfs.server.namenode.ha.TestNNHealthCheck
          hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
          hadoop.hdfs.TestDFSAddressConfig
          hadoop.fs.TestSymlinkHdfsFileSystem
          hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics
          hadoop.tools.TestJMXGet
          hadoop.hdfs.server.namenode.TestEditLogJournalFailures
          hadoop.hdfs.TestDFSMkdirs
          hadoop.hdfs.TestWriteRead
          hadoop.hdfs.TestClientBlockVerification
          hadoop.fs.viewfs.TestViewFsDefaultValue
          hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles
          hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages
          hadoop.hdfs.TestDatanodeConfig
          hadoop.hdfs.server.namenode.TestNameNodeResourceChecker
          hadoop.hdfs.server.datanode.TestDataNodeECN
          hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate
          hadoop.hdfs.server.datanode.TestDataNodeMXBean
          hadoop.hdfs.server.namenode.TestAclConfigFlag
          hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd
          hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA
          hadoop.hdfs.server.namenode.TestFsLimits
          hadoop.hdfs.server.namenode.TestBackupNode
          hadoop.cli.TestHDFSCLI
          hadoop.hdfs.server.namenode.ha.TestStandbyIsHot
          hadoop.hdfs.server.namenode.TestAllowFormat
          hadoop.hdfs.server.namenode.TestValidateConfigurationSettings
          hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant
          hadoop.hdfs.server.namenode.TestNameNodeRpcServer
          hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
          hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport
          hadoop.fs.contract.hdfs.TestHDFSContractDelete
          hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
          hadoop.hdfs.server.namenode.TestCheckpoint
          hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing
          hadoop.hdfs.server.namenode.TestDeadDatanode
          hadoop.hdfs.TestDatanodeReport
          hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          hadoop.hdfs.server.namenode.TestAuditLogger
          hadoop.hdfs.server.namenode.ha.TestGetGroupsWithHA
          hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
          hadoop.hdfs.server.datanode.TestDatanodeProtocolRetryPolicy
          hadoop.hdfs.TestDFSShellGenericOptions
          hadoop.security.TestPermission
          hadoop.hdfs.TestMissingBlocksAlert
          hadoop.fs.viewfs.TestViewFsHdfs
          hadoop.hdfs.TestClose
          hadoop.cli.TestDeleteCLI
          hadoop.hdfs.tools.TestDebugAdmin
          hadoop.hdfs.TestFileCorruption
          hadoop.hdfs.server.namenode.TestFsckWithMultipleNameNodes
          hadoop.fs.TestFcHdfsCreateMkdir
          hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots
          hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing
          hadoop.hdfs.server.datanode.TestDataNodeUUID
          hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
          hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
          hadoop.hdfs.server.namenode.TestXAttrConfigFlag
          hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
          hadoop.fs.permission.TestStickyBit
          hadoop.hdfs.client.impl.TestLeaseRenewer
          hadoop.hdfs.tools.TestDFSAdminWithHA
          hadoop.fs.viewfs.TestViewFsWithAcls
          hadoop.hdfs.tools.TestStoragePolicyCommands
          hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
          hadoop.hdfs.server.datanode.TestDatanodeStartupOptions
          hadoop.hdfs.server.namenode.TestStartup
          hadoop.hdfs.TestMultiThreadedHflush
          hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
          hadoop.TestGenericRefresh
          hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
          hadoop.hdfs.server.datanode.TestBlockReplacement
          hadoop.hdfs.server.datanode.TestBlockPoolManager
          hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
          hadoop.hdfs.TestQuota
          hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
          hadoop.hdfs.security.TestDelegationTokenForProxyUser
          hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
          hadoop.fs.viewfs.TestViewFsAtHdfsRoot
          hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling
          hadoop.hdfs.server.blockmanagement.TestPendingReplication
          hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion
          hadoop.hdfs.TestCrcCorruption
          hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
          hadoop.tracing.TestTracingShortCircuitLocalRead
          hadoop.fs.TestFcHdfsPermission
          hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
          hadoop.hdfs.TestLeaseRecovery
          hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics
          hadoop.fs.contract.hdfs.TestHDFSContractRootDirectory
          hadoop.fs.contract.hdfs.TestHDFSContractCreate
          hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN
          hadoop.hdfs.TestFileAppend
          hadoop.hdfs.TestGetBlocks
          hadoop.hdfs.web.TestWebHDFSOAuth2
          hadoop.hdfs.server.namenode.TestProcessCorruptBlocks
          hadoop.fs.TestResolveHdfsSymlink
          hadoop.hdfs.server.datanode.TestStorageReport
          hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl
          hadoop.hdfs.TestDFSUpgrade
          hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
          hadoop.hdfs.server.namenode.TestCacheDirectives
          hadoop.hdfs.server.namenode.TestNameNodeMXBean
          hadoop.hdfs.server.namenode.TestLargeDirectoryDelete
          hadoop.hdfs.server.namenode.TestFSImageWithSnapshot
          hadoop.tracing.TestTraceAdmin
          hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap
          hadoop.hdfs.server.namenode.TestGetImageServlet
          hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
          hadoop.hdfs.server.namenode.snapshot.TestSnapshotRename
          hadoop.hdfs.server.namenode.TestNameEditsConfigs
          hadoop.cli.TestCacheAdminCLI
          hadoop.hdfs.server.datanode.TestDataNodeExit
          hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
          hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication
          hadoop.hdfs.server.datanode.TestRefreshNamenodes
          hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
          hadoop.hdfs.server.namenode.TestHostsFiles
          hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
          hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots
          hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
          hadoop.hdfs.server.namenode.TestEditLogRace
          hadoop.hdfs.server.namenode.ha.TestHAConfiguration
          hadoop.hdfs.server.namenode.TestFSNamesystem
          hadoop.TestRefreshCallQueue
          hadoop.hdfs.TestSetTimes
          hadoop.hdfs.TestDFSUpgradeFromImage
          hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica
          hadoop.hdfs.server.namenode.ha.TestFailoverWithBlockTokensEnabled
          hadoop.hdfs.server.datanode.TestDataNodeMetrics
          hadoop.fs.TestEnhancedByteBufferAccess
          hadoop.cli.TestCryptoAdminCLI
          hadoop.hdfs.TestClientProtocolForPipelineRecovery
          hadoop.hdfs.TestDFSStorageStateRecovery
          hadoop.fs.contract.hdfs.TestHDFSContractGetFileStatus
          hadoop.hdfs.server.namenode.TestDefaultBlockPlacementPolicy
          hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
          hadoop.hdfs.TestBlockMissingException
          hadoop.hdfs.server.namenode.TestSnapshotPathINodes
          hadoop.hdfs.server.namenode.TestLeaseManager
          hadoop.hdfs.server.datanode.web.dtp.TestDtpHttp2
          hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade
          hadoop.fs.viewfs.TestViewFileSystemHdfs
          hadoop.hdfs.server.datanode.TestBlockRecovery
          hadoop.hdfs.tools.TestDFSHAAdminMiniCluster
          hadoop.hdfs.security.TestDelegationToken
          hadoop.hdfs.server.namenode.TestBlockUnderConstruction
          hadoop.hdfs.TestFileConcurrentReader
          hadoop.hdfs.TestDFSClientRetries
          hadoop.hdfs.server.namenode.TestINodeAttributeProvider
          hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens
          hadoop.hdfs.tools.TestDFSAdmin
          hadoop.hdfs.TestRead
          hadoop.hdfs.TestDFSInotifyEventInputStream
          hadoop.hdfs.server.datanode.TestDataNodeInitStorage
          hadoop.hdfs.server.namenode.TestFSNamesystemMBean
          hadoop.fs.loadGenerator.TestLoadGenerator
          hadoop.fs.contract.hdfs.TestHDFSContractAppend
          hadoop.hdfs.server.namenode.TestSecondaryWebUi
          hadoop.fs.viewfs.TestViewFileSystemWithAcls
          hadoop.hdfs.server.datanode.TestIncrementalBlockReports
          hadoop.hdfs.TestSetrepIncreasing
          hadoop.fs.TestFcHdfsSetUMask
          hadoop.hdfs.TestLargeBlock
          hadoop.hdfs.server.namenode.ha.TestLossyRetryInvocationHandler
          hadoop.hdfs.TestFileAppend3
          hadoop.hdfs.server.blockmanagement.TestSequentialBlockId
          hadoop.hdfs.TestDFSClientFailover
          hadoop.hdfs.server.namenode.TestGetBlockLocations
          hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
          hadoop.hdfs.server.namenode.snapshot.TestSnapshotStatsMXBean
          hadoop.hdfs.server.datanode.TestReadOnlySharedStorage
          hadoop.hdfs.server.namenode.ha.TestHAMetrics
          hadoop.hdfs.server.namenode.TestFSDirectory
          hadoop.hdfs.server.namenode.TestEditLogAutoroll
          hadoop.hdfs.server.namenode.TestFSImage
          hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy
          hadoop.hdfs.server.namenode.TestAddBlockRetry
          hadoop.hdfs.server.namenode.ha.TestRemoteNameNodeInfo
          hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.TestPipelines
          hadoop.hdfs.TestAppendDifferentChecksum
          hadoop.hdfs.server.namenode.TestTransferFsImage
          hadoop.hdfs.server.namenode.ha.TestHAFsck
          hadoop.hdfs.TestSeekBug
          hadoop.hdfs.TestFileAppend4
          hadoop.hdfs.TestDFSRemove
          hadoop.hdfs.server.blockmanagement.TestBlockReportRateLimiting
          hadoop.hdfs.server.blockmanagement.TestReplicationPolicyConsiderLoad
          hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
          hadoop.hdfs.TestBlockReaderLocalLegacy
          hadoop.hdfs.TestBlocksScheduledCounter
          hadoop.hdfs.TestExternalBlockReader
          hadoop.fs.viewfs.TestViewFsFileStatusHdfs
          hadoop.hdfs.TestSnapshotCommands
          hadoop.hdfs.server.datanode.TestDataNodeTransferSocketSize
          hadoop.fs.shell.TestHdfsTextCommand
          hadoop.hdfs.TestWriteConfigurationToDFS
          hadoop.hdfs.TestDFSRollback
          hadoop.hdfs.server.namenode.snapshot.TestGetContentSummaryWithSnapshot
          hadoop.hdfs.TestFSInputChecker
          hadoop.hdfs.server.namenode.TestClusterId
          hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement
          hadoop.fs.TestGlobPaths
          hadoop.hdfs.TestDataTransferKeepalive
          hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM
          hadoop.hdfs.TestHDFSTrash
          hadoop.hdfs.server.namenode.TestNameNodeAcl
          hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation
          hadoop.fs.contract.hdfs.TestHDFSContractRename
          hadoop.hdfs.TestHDFSServerPorts
          hadoop.hdfs.TestRenameWhileOpen
          hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot
          hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol
          hadoop.hdfs.server.datanode.TestDataNodeMetricsLogger
          hadoop.hdfs.TestHDFSFileSystemContract
          hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForXAttr
          hadoop.net.TestNetworkTopology
          hadoop.hdfs.security.token.block.TestBlockToken
          hadoop.hdfs.server.datanode.TestIncrementalBrVariations
          hadoop.hdfs.server.datanode.TestHSync
          hadoop.hdfs.server.namenode.TestNamenodeRetryCache
          hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary
          hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
          hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
          hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
          hadoop.hdfs.server.namenode.ha.TestDNFencing
          hadoop.hdfs.server.namenode.TestCreateEditsLog
          hadoop.hdfs.server.namenode.TestFileContextXAttr



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12761170/HDFS-9107.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 88d89267
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12545/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12545/testReport/
        Java 1.7.0_55
        uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12545/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 50s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 56s There were no new javac warning messages. +1 javadoc 10m 18s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 22s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 12s Pre-build of native portion -1 hdfs tests 42m 47s Tests failed in hadoop-hdfs.     88m 26s   Reason Tests Failed unit tests hadoop.hdfs.TestFileStatus   hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes   hadoop.hdfs.server.namenode.TestINodeFile   hadoop.fs.contract.hdfs.TestHDFSContractOpen   hadoop.hdfs.server.datanode.TestFsDatasetCache   hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart   hadoop.hdfs.TestFileCreationDelete   hadoop.hdfs.server.namenode.ha.TestHASafeMode   hadoop.hdfs.TestEncryptionZonesWithHA   hadoop.fs.contract.hdfs.TestHDFSContractMkdir   hadoop.hdfs.server.namenode.TestNameNodeXAttr   hadoop.hdfs.server.namenode.ha.TestBootstrapStandby   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestDecommission   hadoop.hdfs.server.namenode.TestFSEditLogLoader   hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA   hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages   hadoop.hdfs.server.datanode.TestCachingStrategy   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.cli.TestXAttrCLI   hadoop.hdfs.server.namenode.TestDeleteRace   hadoop.hdfs.server.namenode.TestParallelImageWrite   hadoop.hdfs.server.namenode.TestNameNodeRespectsBindHostKeys   hadoop.hdfs.server.namenode.TestNNStorageRetentionFunctional   hadoop.hdfs.server.namenode.TestSaveNamespace   hadoop.hdfs.TestDFSRename   hadoop.hdfs.util.TestDiff   hadoop.hdfs.server.datanode.TestDataNodeFSDataSetSink   hadoop.hdfs.server.namenode.TestFsck   hadoop.hdfs.server.namenode.ha.TestHarFileSystemWithHA   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration   hadoop.hdfs.server.datanode.TestDeleteBlockPool   hadoop.hdfs.TestRemoteBlockReader2   hadoop.hdfs.server.namenode.TestStorageRestore   hadoop.hdfs.server.namenode.TestFileLimit   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.fs.contract.hdfs.TestHDFSContractSetTimes   hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots   hadoop.hdfs.qjournal.TestNNWithQJM   hadoop.hdfs.server.namenode.TestFSPermissionChecker   hadoop.hdfs.server.namenode.TestSecureNameNode   hadoop.hdfs.server.namenode.TestFileContextAcl   hadoop.hdfs.server.datanode.TestDataXceiverLazyPersistHint   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.TestDataTransferProtocol   hadoop.fs.viewfs.TestViewFsWithXAttrs   hadoop.hdfs.server.datanode.TestDiskError   hadoop.hdfs.TestFsShellPermission   hadoop.hdfs.server.namenode.TestMalformedURLs   hadoop.hdfs.TestReadWhileWriting   hadoop.fs.TestSWebHdfsFileContextMainOperations   hadoop.hdfs.TestIsMethodSupported   hadoop.hdfs.TestParallelShortCircuitReadNoChecksum   hadoop.hdfs.server.blockmanagement.TestAvailableSpaceBlockPlacementPolicy   hadoop.hdfs.TestFileCreationClient   hadoop.cli.TestAclCLI   hadoop.hdfs.TestFSOutputSummer   hadoop.fs.TestHDFSFileContextMainOperations   hadoop.hdfs.TestDFSStartupVersions   hadoop.hdfs.server.namenode.ha.TestFailureOfSharedDir   hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters   hadoop.hdfs.TestParallelShortCircuitLegacyRead   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.server.namenode.ha.TestStateTransitionFailure   hadoop.hdfs.server.namenode.TestGenericJournalConf   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.TestFileAppend2   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory   hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup   hadoop.hdfs.server.datanode.TestSimulatedFSDataset   hadoop.hdfs.server.datanode.TestTransferRbw   hadoop.fs.contract.hdfs.TestHDFSContractSeek   hadoop.hdfs.TestLocalDFS   hadoop.hdfs.server.mover.TestStorageMover   hadoop.hdfs.server.datanode.TestDataStorage   hadoop.hdfs.server.namenode.TestTruncateQuotaUpdate   hadoop.hdfs.server.namenode.TestMetadataVersionOutput   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.server.namenode.TestFSImageWithAcl   hadoop.fs.TestUnbuffer   hadoop.hdfs.TestDFSPermission   hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks   hadoop.hdfs.server.namenode.TestAddBlock   hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles   hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold   hadoop.hdfs.server.namenode.TestMetaSave   hadoop.hdfs.TestSmallBlock   hadoop.hdfs.tools.TestDFSHAAdmin   hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks   hadoop.tracing.TestTracing   hadoop.hdfs.server.datanode.TestBPOfferService   hadoop.hdfs.server.namenode.TestCommitBlockSynchronization   hadoop.hdfs.server.datanode.TestDatanodeRegister   hadoop.hdfs.server.datanode.TestDataDirs   hadoop.hdfs.server.datanode.TestTriggerBlockReport   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter   hadoop.fs.contract.hdfs.TestHDFSContractConcat   hadoop.hdfs.TestReplication   hadoop.hdfs.server.namenode.TestEditLog   hadoop.security.TestRefreshUserMappings   hadoop.hdfs.server.namenode.snapshot.TestFileContextSnapshot   hadoop.hdfs.server.mover.TestMover   hadoop.hdfs.tools.TestDFSZKFailoverController   hadoop.hdfs.TestDefaultNameNodePort   hadoop.hdfs.server.namenode.TestHDFSConcat   hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS   hadoop.hdfs.server.namenode.TestFSImageWithXAttr   hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer   hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs   hadoop.security.TestPermissionSymlinks   hadoop.hdfs.TestHFlush   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.fs.TestSymlinkHdfsDisable   hadoop.hdfs.server.namenode.TestQuotaByStorageType   hadoop.hdfs.TestFileCreation   hadoop.hdfs.server.namenode.snapshot.TestDisallowModifyROSnapshot   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.server.namenode.ha.TestQuotasWithHA   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits   hadoop.hdfs.server.namenode.ha.TestHAStateTransitions   hadoop.hdfs.server.namenode.snapshot.TestSetQuotaWithSnapshot   hadoop.hdfs.server.namenode.metrics.TestNNMetricFilesInGetListingOps   hadoop.hdfs.TestRollingUpgradeDowngrade   hadoop.hdfs.server.namenode.ha.TestEditLogsDuringFailover   hadoop.hdfs.TestBlockReaderFactory   hadoop.hdfs.TestDFSOutputStream   hadoop.fs.TestUrlStreamHandler   hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot   hadoop.hdfs.TestModTime   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.fs.viewfs.TestViewFileSystemWithXAttrs   hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.server.namenode.TestNameNodeRecovery   hadoop.hdfs.TestRestartDFS   hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade   hadoop.hdfs.TestInjectionForSimulatedStorage   hadoop.hdfs.TestMiniDFSCluster   hadoop.hdfs.server.datanode.TestFsDatasetCacheRevocation   hadoop.hdfs.server.namenode.snapshot.TestSnapshot   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.namenode.TestProtectedDirectories   hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot   hadoop.hdfs.server.namenode.TestNameNodeMetricsLogger   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.namenode.TestAuditLogs   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.fs.TestWebHdfsFileContextMainOperations   hadoop.hdfs.server.namenode.ha.TestNNHealthCheck   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.TestDFSAddressConfig   hadoop.fs.TestSymlinkHdfsFileSystem   hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics   hadoop.tools.TestJMXGet   hadoop.hdfs.server.namenode.TestEditLogJournalFailures   hadoop.hdfs.TestDFSMkdirs   hadoop.hdfs.TestWriteRead   hadoop.hdfs.TestClientBlockVerification   hadoop.fs.viewfs.TestViewFsDefaultValue   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles   hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages   hadoop.hdfs.TestDatanodeConfig   hadoop.hdfs.server.namenode.TestNameNodeResourceChecker   hadoop.hdfs.server.datanode.TestDataNodeECN   hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate   hadoop.hdfs.server.datanode.TestDataNodeMXBean   hadoop.hdfs.server.namenode.TestAclConfigFlag   hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd   hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA   hadoop.hdfs.server.namenode.TestFsLimits   hadoop.hdfs.server.namenode.TestBackupNode   hadoop.cli.TestHDFSCLI   hadoop.hdfs.server.namenode.ha.TestStandbyIsHot   hadoop.hdfs.server.namenode.TestAllowFormat   hadoop.hdfs.server.namenode.TestValidateConfigurationSettings   hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant   hadoop.hdfs.server.namenode.TestNameNodeRpcServer   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport   hadoop.fs.contract.hdfs.TestHDFSContractDelete   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery   hadoop.hdfs.server.namenode.TestCheckpoint   hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing   hadoop.hdfs.server.namenode.TestDeadDatanode   hadoop.hdfs.TestDatanodeReport   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.namenode.TestAuditLogger   hadoop.hdfs.server.namenode.ha.TestGetGroupsWithHA   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.server.datanode.TestDatanodeProtocolRetryPolicy   hadoop.hdfs.TestDFSShellGenericOptions   hadoop.security.TestPermission   hadoop.hdfs.TestMissingBlocksAlert   hadoop.fs.viewfs.TestViewFsHdfs   hadoop.hdfs.TestClose   hadoop.cli.TestDeleteCLI   hadoop.hdfs.tools.TestDebugAdmin   hadoop.hdfs.TestFileCorruption   hadoop.hdfs.server.namenode.TestFsckWithMultipleNameNodes   hadoop.fs.TestFcHdfsCreateMkdir   hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots   hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing   hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer   hadoop.hdfs.server.namenode.TestXAttrConfigFlag   hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication   hadoop.fs.permission.TestStickyBit   hadoop.hdfs.client.impl.TestLeaseRenewer   hadoop.hdfs.tools.TestDFSAdminWithHA   hadoop.fs.viewfs.TestViewFsWithAcls   hadoop.hdfs.tools.TestStoragePolicyCommands   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.server.datanode.TestDatanodeStartupOptions   hadoop.hdfs.server.namenode.TestStartup   hadoop.hdfs.TestMultiThreadedHflush   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits   hadoop.TestGenericRefresh   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.datanode.TestBlockPoolManager   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.TestQuota   hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean   hadoop.fs.viewfs.TestViewFsAtHdfsRoot   hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling   hadoop.hdfs.server.blockmanagement.TestPendingReplication   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.namenode.TestSecurityTokenEditLog   hadoop.tracing.TestTracingShortCircuitLocalRead   hadoop.fs.TestFcHdfsPermission   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy   hadoop.hdfs.TestLeaseRecovery   hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics   hadoop.fs.contract.hdfs.TestHDFSContractRootDirectory   hadoop.fs.contract.hdfs.TestHDFSContractCreate   hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN   hadoop.hdfs.TestFileAppend   hadoop.hdfs.TestGetBlocks   hadoop.hdfs.web.TestWebHDFSOAuth2   hadoop.hdfs.server.namenode.TestProcessCorruptBlocks   hadoop.fs.TestResolveHdfsSymlink   hadoop.hdfs.server.datanode.TestStorageReport   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl   hadoop.hdfs.TestDFSUpgrade   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.server.namenode.TestCacheDirectives   hadoop.hdfs.server.namenode.TestNameNodeMXBean   hadoop.hdfs.server.namenode.TestLargeDirectoryDelete   hadoop.hdfs.server.namenode.TestFSImageWithSnapshot   hadoop.tracing.TestTraceAdmin   hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap   hadoop.hdfs.server.namenode.TestGetImageServlet   hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer   hadoop.hdfs.server.namenode.snapshot.TestSnapshotRename   hadoop.hdfs.server.namenode.TestNameEditsConfigs   hadoop.cli.TestCacheAdminCLI   hadoop.hdfs.server.datanode.TestDataNodeExit   hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage   hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication   hadoop.hdfs.server.datanode.TestRefreshNamenodes   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport   hadoop.hdfs.server.namenode.TestHostsFiles   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots   hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength   hadoop.hdfs.server.namenode.TestEditLogRace   hadoop.hdfs.server.namenode.ha.TestHAConfiguration   hadoop.hdfs.server.namenode.TestFSNamesystem   hadoop.TestRefreshCallQueue   hadoop.hdfs.TestSetTimes   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica   hadoop.hdfs.server.namenode.ha.TestFailoverWithBlockTokensEnabled   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.fs.TestEnhancedByteBufferAccess   hadoop.cli.TestCryptoAdminCLI   hadoop.hdfs.TestClientProtocolForPipelineRecovery   hadoop.hdfs.TestDFSStorageStateRecovery   hadoop.fs.contract.hdfs.TestHDFSContractGetFileStatus   hadoop.hdfs.server.namenode.TestDefaultBlockPlacementPolicy   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.TestBlockMissingException   hadoop.hdfs.server.namenode.TestSnapshotPathINodes   hadoop.hdfs.server.namenode.TestLeaseManager   hadoop.hdfs.server.datanode.web.dtp.TestDtpHttp2   hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade   hadoop.fs.viewfs.TestViewFileSystemHdfs   hadoop.hdfs.server.datanode.TestBlockRecovery   hadoop.hdfs.tools.TestDFSHAAdminMiniCluster   hadoop.hdfs.security.TestDelegationToken   hadoop.hdfs.server.namenode.TestBlockUnderConstruction   hadoop.hdfs.TestFileConcurrentReader   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.namenode.TestINodeAttributeProvider   hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens   hadoop.hdfs.tools.TestDFSAdmin   hadoop.hdfs.TestRead   hadoop.hdfs.TestDFSInotifyEventInputStream   hadoop.hdfs.server.datanode.TestDataNodeInitStorage   hadoop.hdfs.server.namenode.TestFSNamesystemMBean   hadoop.fs.loadGenerator.TestLoadGenerator   hadoop.fs.contract.hdfs.TestHDFSContractAppend   hadoop.hdfs.server.namenode.TestSecondaryWebUi   hadoop.fs.viewfs.TestViewFileSystemWithAcls   hadoop.hdfs.server.datanode.TestIncrementalBlockReports   hadoop.hdfs.TestSetrepIncreasing   hadoop.fs.TestFcHdfsSetUMask   hadoop.hdfs.TestLargeBlock   hadoop.hdfs.server.namenode.ha.TestLossyRetryInvocationHandler   hadoop.hdfs.TestFileAppend3   hadoop.hdfs.server.blockmanagement.TestSequentialBlockId   hadoop.hdfs.TestDFSClientFailover   hadoop.hdfs.server.namenode.TestGetBlockLocations   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.server.namenode.snapshot.TestSnapshotStatsMXBean   hadoop.hdfs.server.datanode.TestReadOnlySharedStorage   hadoop.hdfs.server.namenode.ha.TestHAMetrics   hadoop.hdfs.server.namenode.TestFSDirectory   hadoop.hdfs.server.namenode.TestEditLogAutoroll   hadoop.hdfs.server.namenode.TestFSImage   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy   hadoop.hdfs.server.namenode.TestAddBlockRetry   hadoop.hdfs.server.namenode.ha.TestRemoteNameNodeInfo   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.TestPipelines   hadoop.hdfs.TestAppendDifferentChecksum   hadoop.hdfs.server.namenode.TestTransferFsImage   hadoop.hdfs.server.namenode.ha.TestHAFsck   hadoop.hdfs.TestSeekBug   hadoop.hdfs.TestFileAppend4   hadoop.hdfs.TestDFSRemove   hadoop.hdfs.server.blockmanagement.TestBlockReportRateLimiting   hadoop.hdfs.server.blockmanagement.TestReplicationPolicyConsiderLoad   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer   hadoop.hdfs.TestBlockReaderLocalLegacy   hadoop.hdfs.TestBlocksScheduledCounter   hadoop.hdfs.TestExternalBlockReader   hadoop.fs.viewfs.TestViewFsFileStatusHdfs   hadoop.hdfs.TestSnapshotCommands   hadoop.hdfs.server.datanode.TestDataNodeTransferSocketSize   hadoop.fs.shell.TestHdfsTextCommand   hadoop.hdfs.TestWriteConfigurationToDFS   hadoop.hdfs.TestDFSRollback   hadoop.hdfs.server.namenode.snapshot.TestGetContentSummaryWithSnapshot   hadoop.hdfs.TestFSInputChecker   hadoop.hdfs.server.namenode.TestClusterId   hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement   hadoop.fs.TestGlobPaths   hadoop.hdfs.TestDataTransferKeepalive   hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM   hadoop.hdfs.TestHDFSTrash   hadoop.hdfs.server.namenode.TestNameNodeAcl   hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation   hadoop.fs.contract.hdfs.TestHDFSContractRename   hadoop.hdfs.TestHDFSServerPorts   hadoop.hdfs.TestRenameWhileOpen   hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot   hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol   hadoop.hdfs.server.datanode.TestDataNodeMetricsLogger   hadoop.hdfs.TestHDFSFileSystemContract   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForXAttr   hadoop.net.TestNetworkTopology   hadoop.hdfs.security.token.block.TestBlockToken   hadoop.hdfs.server.datanode.TestIncrementalBrVariations   hadoop.hdfs.server.datanode.TestHSync   hadoop.hdfs.server.namenode.TestNamenodeRetryCache   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary   hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup   hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality   hadoop.hdfs.server.namenode.TestListCorruptFileBlocks   hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.server.namenode.TestCreateEditsLog   hadoop.hdfs.server.namenode.TestFileContextXAttr Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12761170/HDFS-9107.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 88d89267 hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12545/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12545/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12545/console This message was automatically generated.
        Hide
        stevel@apache.org Steve Loughran added a comment -
        1. I don't trust monotonicNow if the thread can suspend between calls; cores on different sockets may give different answers, though it's not something I've seen in the field.
        2. kicking off a new test run; this looks like an mvn dependency failure
        Show
        stevel@apache.org Steve Loughran added a comment - I don't trust monotonicNow if the thread can suspend between calls; cores on different sockets may give different answers, though it's not something I've seen in the field. kicking off a new test run; this looks like an mvn dependency failure
        Hide
        hitliuyi Yi Liu added a comment -

        Thanks Daryn Sharp, the issue seems critical.
        I have few comments:
        1. the default heartbeat recheck interval is 5 minutes if not configured, is it possible a full GC longer than 5 minutes? I see some full gc lasts tens of seconds, but not saw so long, of course, it depends on the heap size (old generation). Actually the data node dead (heartbeat expire) interval is 2x than heartbeat recheck interval, so the full gc should last 10 minutes.

        2. The patch assumes the full gc happens during the sleep, it's most possible, but if it happens after long now = .. or setting lastHeatbeatCheck to now, the issue still exists, even though small probability.

        But I would like to give +1 for the patch, since it solves the issue if really happen, and doesn't affect existing logic.

        Show
        hitliuyi Yi Liu added a comment - Thanks Daryn Sharp , the issue seems critical. I have few comments: 1. the default heartbeat recheck interval is 5 minutes if not configured, is it possible a full GC longer than 5 minutes? I see some full gc lasts tens of seconds, but not saw so long, of course, it depends on the heap size (old generation). Actually the data node dead (heartbeat expire) interval is 2x than heartbeat recheck interval, so the full gc should last 10 minutes. 2. The patch assumes the full gc happens during the sleep , it's most possible, but if it happens after long now = .. or setting lastHeatbeatCheck to now , the issue still exists, even though small probability. But I would like to give +1 for the patch, since it solves the issue if really happen, and doesn't affect existing logic.
        Hide
        hitliuyi Yi Liu added a comment - - edited

        Sorry I just see Steve's comments.

        cores on different sockets may give different answers

        About the nanoTime, yes, I also ever saw similar points and discussion like this, but seems it's not correct and nanoTime is safe, see more discussion in http://stackoverflow.com/questions/510462/is-system-nanotime-completely-useless. (There are some links to oracle article in the discussion.)

        Show
        hitliuyi Yi Liu added a comment - - edited Sorry I just see Steve's comments. cores on different sockets may give different answers About the nanoTime , yes, I also ever saw similar points and discussion like this, but seems it's not correct and nanoTime is safe, see more discussion in http://stackoverflow.com/questions/510462/is-system-nanotime-completely-useless . (There are some links to oracle article in the discussion.)
        Hide
        daryn Daryn Sharp added a comment -

        Yi Liu, good points.

        1. Trust me, it's more than possible for a ~10 min full GC with a big heap. We've even bumped the recheck up on the largest clusters. I should mention these big clusters go through 2-4 full GCs at startup while loading... The overhead of artificially losing nodes doesn't help. This patch won't stop a full GC during image load, or the first full GC in safemode, but should reduce the probability of additional full GCs.
        2. I thought of the exact same thing this weekend. I'll post a revised and equally small patch that addresses the issue more thoroughly.
        Show
        daryn Daryn Sharp added a comment - Yi Liu , good points. Trust me, it's more than possible for a ~10 min full GC with a big heap. We've even bumped the recheck up on the largest clusters. I should mention these big clusters go through 2-4 full GCs at startup while loading... The overhead of artificially losing nodes doesn't help. This patch won't stop a full GC during image load, or the first full GC in safemode, but should reduce the probability of additional full GCs. I thought of the exact same thing this weekend. I'll post a revised and equally small patch that addresses the issue more thoroughly.
        Hide
        cmccabe Colin P. McCabe added a comment -

        I don't trust monotonicNow if the thread can suspend between calls; cores on different sockets may give different answers, though it's not something I've seen in the field.

        Oracle's blog here [ https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks ] says:

        If you are interested in measuring/calculating elapsed time, then always use System.nanoTime(). On most systems it will give a resolution on the order of microseconds. Be aware though, this call can also take microseconds to execute on some platforms.

        Of course, System#nanoTime is just a very thin wrapper around the operating system's monotonic clock. In x86-land, the monotonic clock generally comes from one of two sources: the TSC (timestamp counter) or the HPET (high precision event timer).

        In the 2000s, the TSC started becoming less useful because multi-core systems started becoming more common, and at that time, TSC wasn't synchronized across cores. This has since changed (at least for Intel systems), and the TSC is now synchronized across cores. So the alarm you are raising is about 5 years too late. Anyway, if you have a "bad" TSC, you can still get System#nanoTime to behave correctly by switching your operating system's clock source to the HPET. It's slower, but more reliable.

        If you want to read more about this, check out https://software.intel.com/en-us/forums/intel-isa-extensions/topic/332570

        tl;dr
        1. Operating systems implement various tricks to work around TSC bad behaviors
        2. TSC bad behaviors are becoming less common in modern CPUs
        3. You don't have to use the TSC if you don't want to!

        Let's let the hardware and OS people do their job and just do ours.

        I agree with Yi Liu... +1 for the patch. Would be even better if we could close that small window of a GC happening at a time other than during the Thread#sleep.

        Show
        cmccabe Colin P. McCabe added a comment - I don't trust monotonicNow if the thread can suspend between calls; cores on different sockets may give different answers, though it's not something I've seen in the field. Oracle's blog here [ https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks ] says: If you are interested in measuring/calculating elapsed time, then always use System.nanoTime(). On most systems it will give a resolution on the order of microseconds. Be aware though, this call can also take microseconds to execute on some platforms. Of course, System#nanoTime is just a very thin wrapper around the operating system's monotonic clock. In x86-land, the monotonic clock generally comes from one of two sources: the TSC (timestamp counter) or the HPET (high precision event timer). In the 2000s, the TSC started becoming less useful because multi-core systems started becoming more common, and at that time, TSC wasn't synchronized across cores. This has since changed (at least for Intel systems), and the TSC is now synchronized across cores. So the alarm you are raising is about 5 years too late. Anyway, if you have a "bad" TSC, you can still get System#nanoTime to behave correctly by switching your operating system's clock source to the HPET. It's slower, but more reliable. If you want to read more about this, check out https://software.intel.com/en-us/forums/intel-isa-extensions/topic/332570 tl;dr 1. Operating systems implement various tricks to work around TSC bad behaviors 2. TSC bad behaviors are becoming less common in modern CPUs 3. You don't have to use the TSC if you don't want to! Let's let the hardware and OS people do their job and just do ours. I agree with Yi Liu ... +1 for the patch. Would be even better if we could close that small window of a GC happening at a time other than during the Thread#sleep .
        Hide
        daryn Daryn Sharp added a comment -

        Use a stopwatch to abort processing in the inner heartbeat checking loop, and then check at end of the entire scan for whether to skip next scan. Even added a meager test.

        Show
        daryn Daryn Sharp added a comment - Use a stopwatch to abort processing in the inner heartbeat checking loop, and then check at end of the entire scan for whether to skip next scan. Even added a meager test.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 17m 54s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 8m 21s There were no new javac warning messages.
        +1 javadoc 10m 25s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 22s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 41s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 16s Pre-build of native portion
        -1 hdfs tests 198m 6s Tests failed in hadoop-hdfs.
            244m 36s  



        Reason Tests
        Failed unit tests hadoop.hdfs.TestReplaceDatanodeOnFailure



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12761485/HDFS-9107.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / b00392d
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12574/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12574/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12574/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 54s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 21s There were no new javac warning messages. +1 javadoc 10m 25s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 22s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 41s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 16s Pre-build of native portion -1 hdfs tests 198m 6s Tests failed in hadoop-hdfs.     244m 36s   Reason Tests Failed unit tests hadoop.hdfs.TestReplaceDatanodeOnFailure Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12761485/HDFS-9107.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / b00392d hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12574/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12574/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12574/console This message was automatically generated.
        Hide
        cmccabe Colin P. McCabe added a comment -

        I guess if we want to be 100% correct, we have to do the stopwatch check right after getting back a "true" result from DatanodeManager#isDatanodeDead, right? Otherwise we could always have a TOCTOU where we have a long GC pause right before calling that function. What do you think?

        Show
        cmccabe Colin P. McCabe added a comment - I guess if we want to be 100% correct, we have to do the stopwatch check right after getting back a "true" result from DatanodeManager#isDatanodeDead , right? Otherwise we could always have a TOCTOU where we have a long GC pause right before calling that function. What do you think?
        Hide
        cmccabe Colin P. McCabe added a comment -

        Also (although I don't feel strongly about this), I don't think we need to optimize by checking at end of the entire scan for whether to skip the next scan. Long GCs are rare enough that we don't need to optimize the code path... just keep it simple.

        Show
        cmccabe Colin P. McCabe added a comment - Also (although I don't feel strongly about this), I don't think we need to optimize by checking at end of the entire scan for whether to skip the next scan. Long GCs are rare enough that we don't need to optimize the code path... just keep it simple.
        Hide
        cmccabe Colin P. McCabe added a comment -

        +1. Test failures not related. We can do cleanups in a follow-on. Thanks, Daryn Sharp

        Show
        cmccabe Colin P. McCabe added a comment - +1. Test failures not related. We can do cleanups in a follow-on. Thanks, Daryn Sharp
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #441 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/441/)
        HDFS-9107. Prevent NN's unrecoverable death spiral after full GC (Daryn Sharp via Colin P. McCabe) (cmccabe: rev 4e7c6a653f108d44589f84d78a03d92ee0e8a3c3)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java
          Add HDFS-9107 to CHANGES.txt (cmccabe: rev 878504dcaacdc1bea42ad571ad5f4e537c1d7167)
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #441 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/441/ ) HDFS-9107 . Prevent NN's unrecoverable death spiral after full GC (Daryn Sharp via Colin P. McCabe) (cmccabe: rev 4e7c6a653f108d44589f84d78a03d92ee0e8a3c3) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java Add HDFS-9107 to CHANGES.txt (cmccabe: rev 878504dcaacdc1bea42ad571ad5f4e537c1d7167) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8521 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8521/)
        HDFS-9107. Prevent NN's unrecoverable death spiral after full GC (Daryn Sharp via Colin P. McCabe) (cmccabe: rev 4e7c6a653f108d44589f84d78a03d92ee0e8a3c3)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java
          Add HDFS-9107 to CHANGES.txt (cmccabe: rev 878504dcaacdc1bea42ad571ad5f4e537c1d7167)
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8521 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8521/ ) HDFS-9107 . Prevent NN's unrecoverable death spiral after full GC (Daryn Sharp via Colin P. McCabe) (cmccabe: rev 4e7c6a653f108d44589f84d78a03d92ee0e8a3c3) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java Add HDFS-9107 to CHANGES.txt (cmccabe: rev 878504dcaacdc1bea42ad571ad5f4e537c1d7167) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #448 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/448/)
        HDFS-9107. Prevent NN's unrecoverable death spiral after full GC (Daryn Sharp via Colin P. McCabe) (cmccabe: rev 4e7c6a653f108d44589f84d78a03d92ee0e8a3c3)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
          Add HDFS-9107 to CHANGES.txt (cmccabe: rev 878504dcaacdc1bea42ad571ad5f4e537c1d7167)
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #448 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/448/ ) HDFS-9107 . Prevent NN's unrecoverable death spiral after full GC (Daryn Sharp via Colin P. McCabe) (cmccabe: rev 4e7c6a653f108d44589f84d78a03d92ee0e8a3c3) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java Add HDFS-9107 to CHANGES.txt (cmccabe: rev 878504dcaacdc1bea42ad571ad5f4e537c1d7167) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #1181 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1181/)
        HDFS-9107. Prevent NN's unrecoverable death spiral after full GC (Daryn Sharp via Colin P. McCabe) (cmccabe: rev 4e7c6a653f108d44589f84d78a03d92ee0e8a3c3)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
          Add HDFS-9107 to CHANGES.txt (cmccabe: rev 878504dcaacdc1bea42ad571ad5f4e537c1d7167)
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #1181 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1181/ ) HDFS-9107 . Prevent NN's unrecoverable death spiral after full GC (Daryn Sharp via Colin P. McCabe) (cmccabe: rev 4e7c6a653f108d44589f84d78a03d92ee0e8a3c3) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java Add HDFS-9107 to CHANGES.txt (cmccabe: rev 878504dcaacdc1bea42ad571ad5f4e537c1d7167) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2359 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2359/)
        HDFS-9107. Prevent NN's unrecoverable death spiral after full GC (Daryn Sharp via Colin P. McCabe) (cmccabe: rev 4e7c6a653f108d44589f84d78a03d92ee0e8a3c3)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
          Add HDFS-9107 to CHANGES.txt (cmccabe: rev 878504dcaacdc1bea42ad571ad5f4e537c1d7167)
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2359 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2359/ ) HDFS-9107 . Prevent NN's unrecoverable death spiral after full GC (Daryn Sharp via Colin P. McCabe) (cmccabe: rev 4e7c6a653f108d44589f84d78a03d92ee0e8a3c3) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java Add HDFS-9107 to CHANGES.txt (cmccabe: rev 878504dcaacdc1bea42ad571ad5f4e537c1d7167) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2386 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2386/)
        HDFS-9107. Prevent NN's unrecoverable death spiral after full GC (Daryn Sharp via Colin P. McCabe) (cmccabe: rev 4e7c6a653f108d44589f84d78a03d92ee0e8a3c3)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java
          Add HDFS-9107 to CHANGES.txt (cmccabe: rev 878504dcaacdc1bea42ad571ad5f4e537c1d7167)
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2386 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2386/ ) HDFS-9107 . Prevent NN's unrecoverable death spiral after full GC (Daryn Sharp via Colin P. McCabe) (cmccabe: rev 4e7c6a653f108d44589f84d78a03d92ee0e8a3c3) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java Add HDFS-9107 to CHANGES.txt (cmccabe: rev 878504dcaacdc1bea42ad571ad5f4e537c1d7167) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #419 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/419/)
        HDFS-9107. Prevent NN's unrecoverable death spiral after full GC (Daryn Sharp via Colin P. McCabe) (cmccabe: rev 4e7c6a653f108d44589f84d78a03d92ee0e8a3c3)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java
          Add HDFS-9107 to CHANGES.txt (cmccabe: rev 878504dcaacdc1bea42ad571ad5f4e537c1d7167)
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #419 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/419/ ) HDFS-9107 . Prevent NN's unrecoverable death spiral after full GC (Daryn Sharp via Colin P. McCabe) (cmccabe: rev 4e7c6a653f108d44589f84d78a03d92ee0e8a3c3) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java Add HDFS-9107 to CHANGES.txt (cmccabe: rev 878504dcaacdc1bea42ad571ad5f4e537c1d7167) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

          People

          • Assignee:
            daryn Daryn Sharp
            Reporter:
            daryn Daryn Sharp
          • Votes:
            0 Vote for this issue
            Watchers:
            25 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development