Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8572

DN always uses HTTP/localhost@REALM principals in SPNEGO

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.1, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In HDFS-7279 the Netty server in DN proxies all servlet requests to the local Jetty instance.

      The Jetty server is configured incorrectly so that it always uses HTTP/locahost@REALM to authenticate spnego requests. As a result, servlets like JMX are no longer accessible in secure deployments.

      1. HDFS-8572.000.patch
        9 kB
        Haohui Mai
      2. HDFS-8572.001.patch
        9 kB
        Haohui Mai
      3. HDFS-8572.002.patch
        9 kB
        Haohui Mai

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 0s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 40s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 11s The applied patch generated 2 new checkstyle issues (total was 154, now 147).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse.
          +1 findbugs 3m 16s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 12s Pre-build of native portion
          -1 hdfs tests 50m 53s Tests failed in hadoop-hdfs.
              97m 26s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestFileStatus
            hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
            hadoop.hdfs.server.namenode.TestINodeFile
            hadoop.fs.contract.hdfs.TestHDFSContractOpen
            hadoop.hdfs.server.datanode.TestFsDatasetCache
            hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart
            hadoop.hdfs.TestFileCreationDelete
            hadoop.hdfs.server.namenode.ha.TestHASafeMode
            hadoop.hdfs.TestEncryptionZonesWithHA
            hadoop.fs.contract.hdfs.TestHDFSContractMkdir
            hadoop.hdfs.TestDFSShell
            hadoop.hdfs.server.namenode.TestNameNodeXAttr
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.TestDecommission
            hadoop.hdfs.TestDatanodeDeath
            hadoop.hdfs.server.namenode.TestFSEditLogLoader
            hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
            hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages
            hadoop.hdfs.server.datanode.TestCachingStrategy
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.cli.TestXAttrCLI
            hadoop.hdfs.server.namenode.TestDeleteRace
            hadoop.hdfs.server.namenode.TestParallelImageWrite
            hadoop.hdfs.server.namenode.TestSaveNamespace
            hadoop.hdfs.TestReplaceDatanodeOnFailure
            hadoop.hdfs.TestDFSRename
            hadoop.hdfs.server.namenode.TestFsck
            hadoop.hdfs.server.namenode.ha.TestHarFileSystemWithHA
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement
            hadoop.hdfs.TestFileCreationEmpty
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration
            hadoop.hdfs.server.datanode.TestDeleteBlockPool
            hadoop.hdfs.TestRemoteBlockReader2
            hadoop.hdfs.server.namenode.TestStorageRestore
            hadoop.hdfs.server.namenode.TestFileLimit
            hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots
            hadoop.hdfs.web.TestWebHdfsFileSystemContract
            hadoop.hdfs.TestDFSFinalize
            hadoop.hdfs.server.namenode.TestFileContextAcl
            hadoop.hdfs.TestDatanodeRegistration
            hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
            hadoop.hdfs.TestDataTransferProtocol
            hadoop.fs.viewfs.TestViewFsWithXAttrs
            hadoop.hdfs.server.datanode.TestDiskError
            hadoop.hdfs.TestFsShellPermission
            hadoop.hdfs.server.namenode.TestMalformedURLs
            hadoop.hdfs.TestBalancerBandwidth
            hadoop.hdfs.TestReadWhileWriting
            hadoop.hdfs.TestDisableConnCache
            hadoop.hdfs.TestIsMethodSupported
            hadoop.hdfs.TestDFSClientExcludedNodes
            hadoop.hdfs.TestParallelShortCircuitReadNoChecksum
            hadoop.hdfs.TestFileCreationClient
            hadoop.cli.TestAclCLI
            hadoop.hdfs.TestFSOutputSummer
            hadoop.fs.TestHDFSFileContextMainOperations
            hadoop.hdfs.TestDFSStartupVersions
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters
            hadoop.hdfs.TestParallelShortCircuitLegacyRead
            hadoop.hdfs.server.blockmanagement.TestBlockManager
            hadoop.hdfs.TestFileAppend2
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory
            hadoop.hdfs.TestParallelUnixDomainRead
            hadoop.hdfs.server.datanode.TestTransferRbw
            hadoop.fs.contract.hdfs.TestHDFSContractSeek
            hadoop.hdfs.TestLocalDFS
            hadoop.hdfs.server.mover.TestStorageMover
            hadoop.hdfs.TestGetFileChecksum
            hadoop.hdfs.server.namenode.TestMetadataVersionOutput
            hadoop.hdfs.server.namenode.ha.TestHAAppend
            hadoop.hdfs.server.namenode.TestFSImageWithAcl
            hadoop.fs.TestUnbuffer
            hadoop.hdfs.TestDFSPermission
            hadoop.hdfs.TestParallelRead
            hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
            hadoop.hdfs.server.namenode.TestAddBlock
            hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles
            hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold
            hadoop.hdfs.server.namenode.TestMetaSave
            hadoop.hdfs.TestSmallBlock
            hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
            hadoop.hdfs.web.TestHttpsFileSystem
            hadoop.tracing.TestTracing
            hadoop.hdfs.TestListFilesInDFS
            hadoop.hdfs.TestFileAppendRestart
            hadoop.hdfs.TestBlockReaderLocal
            hadoop.hdfs.server.datanode.TestTriggerBlockReport
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter
            hadoop.fs.contract.hdfs.TestHDFSContractConcat
            hadoop.hdfs.TestReplication
            hadoop.hdfs.server.namenode.TestEditLog
            hadoop.security.TestRefreshUserMappings
            hadoop.hdfs.server.namenode.snapshot.TestFileContextSnapshot
            hadoop.hdfs.server.mover.TestMover
            hadoop.hdfs.TestSafeMode
            hadoop.hdfs.server.namenode.TestHDFSConcat
            hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
            hadoop.hdfs.server.namenode.TestFSImageWithXAttr
            hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
            hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs
            hadoop.security.TestPermissionSymlinks
            hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.TestHFlush
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.fs.TestSymlinkHdfsDisable
            hadoop.hdfs.server.namenode.TestQuotaByStorageType
            hadoop.hdfs.TestFileCreation
            hadoop.hdfs.server.namenode.snapshot.TestDisallowModifyROSnapshot
            hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
            hadoop.hdfs.server.namenode.ha.TestQuotasWithHA
            hadoop.fs.TestSymlinkHdfsFileContext
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.server.namenode.ha.TestHAStateTransitions
            hadoop.hdfs.server.namenode.snapshot.TestSetQuotaWithSnapshot
            hadoop.hdfs.server.namenode.metrics.TestNNMetricFilesInGetListingOps
            hadoop.hdfs.TestBlockReaderFactory
            hadoop.hdfs.TestRemoteBlockReader
            hadoop.hdfs.TestFetchImage
            hadoop.hdfs.TestDFSOutputStream
            hadoop.fs.TestUrlStreamHandler
            hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot
            hadoop.hdfs.TestModTime
            hadoop.hdfs.TestLeaseRecovery2
            hadoop.fs.viewfs.TestViewFileSystemWithXAttrs
            hadoop.hdfs.TestAppendSnapshotTruncate
            hadoop.hdfs.TestRestartDFS
            hadoop.hdfs.TestDistributedFileSystem
            hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
            hadoop.hdfs.TestInjectionForSimulatedStorage
            hadoop.hdfs.TestMiniDFSCluster
            hadoop.hdfs.server.datanode.TestFsDatasetCacheRevocation
            hadoop.hdfs.server.namenode.snapshot.TestSnapshot
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.TestLease
            hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.server.namenode.TestAuditLogs
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
            hadoop.hdfs.TestDFSAddressConfig
            hadoop.fs.TestSymlinkHdfsFileSystem
            hadoop.hdfs.server.datanode.fsdataset.impl.TestRbwSpaceReservation
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics
            hadoop.tools.TestJMXGet
            hadoop.hdfs.TestDFSMkdirs
            hadoop.hdfs.TestWriteRead
            hadoop.hdfs.TestClientBlockVerification
            hadoop.fs.viewfs.TestViewFsDefaultValue
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles
            hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages
            hadoop.hdfs.TestDatanodeConfig
            hadoop.hdfs.server.namenode.TestNameNodeResourceChecker
            hadoop.hdfs.server.datanode.TestDataNodeECN
            hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate
            hadoop.hdfs.server.datanode.TestDataNodeMXBean
            hadoop.hdfs.server.namenode.TestAclConfigFlag
            hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd
            hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA
            hadoop.hdfs.server.namenode.TestBackupNode
            hadoop.cli.TestHDFSCLI
            hadoop.hdfs.server.namenode.ha.TestStandbyIsHot
            hadoop.hdfs.server.namenode.TestAllowFormat
            hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant
            hadoop.hdfs.server.namenode.TestNameNodeRpcServer
            hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
            hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport
            hadoop.fs.contract.hdfs.TestHDFSContractDelete
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
            hadoop.hdfs.server.namenode.TestCheckpoint
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing
            hadoop.hdfs.server.namenode.TestDeadDatanode
            hadoop.hdfs.web.TestWebHdfsWithAuthenticationFilter
            hadoop.hdfs.TestDatanodeReport
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            hadoop.hdfs.TestClientReportBadBlock
            hadoop.hdfs.server.namenode.TestAuditLogger
            hadoop.hdfs.server.datanode.TestDatanodeProtocolRetryPolicy
            hadoop.hdfs.TestDFSShellGenericOptions
            hadoop.security.TestPermission
            hadoop.hdfs.web.TestWebHDFSAcl
            hadoop.hdfs.TestMissingBlocksAlert
            hadoop.fs.viewfs.TestViewFsHdfs
            hadoop.hdfs.TestClose
            hadoop.hdfs.tools.TestDebugAdmin
            hadoop.hdfs.TestFileCorruption
            hadoop.hdfs.server.namenode.TestFsckWithMultipleNameNodes
            hadoop.fs.TestFcHdfsCreateMkdir
            hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots
            hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing
            hadoop.hdfs.server.datanode.TestDataNodeUUID
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
            hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
            hadoop.hdfs.server.namenode.TestXAttrConfigFlag
            hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
            hadoop.fs.permission.TestStickyBit
            hadoop.fs.viewfs.TestViewFsWithAcls
            hadoop.hdfs.tools.TestStoragePolicyCommands
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.server.namenode.TestStartup
            hadoop.hdfs.TestMultiThreadedHflush
            hadoop.TestGenericRefresh
            hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            hadoop.hdfs.TestQuota
            hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestConnCache
            hadoop.fs.viewfs.TestViewFsAtHdfsRoot
            hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling
            hadoop.hdfs.server.blockmanagement.TestPendingReplication
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
            hadoop.tracing.TestTracingShortCircuitLocalRead
            hadoop.fs.TestFcHdfsPermission
            hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
            hadoop.hdfs.TestLeaseRecovery
            hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics
            hadoop.hdfs.TestFileLengthOnClusterRestart
            hadoop.fs.contract.hdfs.TestHDFSContractRootDirectory
            hadoop.fs.contract.hdfs.TestHDFSContractCreate
            hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.web.TestWebHDFS
            hadoop.hdfs.TestGetBlocks
            hadoop.hdfs.server.namenode.TestProcessCorruptBlocks
            hadoop.fs.TestResolveHdfsSymlink
            hadoop.hdfs.server.datanode.TestStorageReport
            hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl
            hadoop.hdfs.TestDFSUpgrade
            hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
            hadoop.hdfs.TestPread
            hadoop.hdfs.server.namenode.TestCacheDirectives
            hadoop.hdfs.server.namenode.TestNameNodeMXBean
            hadoop.hdfs.server.namenode.TestLargeDirectoryDelete
            hadoop.hdfs.server.namenode.TestFSImageWithSnapshot
            hadoop.tracing.TestTraceAdmin
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap
            hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotRename
            hadoop.hdfs.server.namenode.TestNameEditsConfigs
            hadoop.cli.TestCacheAdminCLI
            hadoop.hdfs.server.datanode.TestDataNodeExit
            hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication
            hadoop.hdfs.server.datanode.TestRefreshNamenodes
            hadoop.hdfs.web.TestWebHDFSForHA
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
            hadoop.hdfs.server.namenode.TestHostsFiles
            hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
            hadoop.hdfs.web.TestFSMainOperationsWebHdfs
            hadoop.hdfs.server.namenode.TestEditLogRace
            hadoop.hdfs.TestReservedRawPaths
            hadoop.TestRefreshCallQueue
            hadoop.hdfs.TestSetTimes
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica
            hadoop.hdfs.server.namenode.ha.TestFailoverWithBlockTokensEnabled
            hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.fs.TestEnhancedByteBufferAccess
            hadoop.cli.TestCryptoAdminCLI
            hadoop.hdfs.TestClientProtocolForPipelineRecovery
            hadoop.hdfs.TestDFSStorageStateRecovery
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.namenode.TestDefaultBlockPlacementPolicy
            hadoop.hdfs.TestListFilesInFileContext
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.TestBlockMissingException
            hadoop.hdfs.server.namenode.TestSnapshotPathINodes
            hadoop.hdfs.server.datanode.web.dtp.TestDtpHttp2
            hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade
            hadoop.fs.viewfs.TestViewFileSystemHdfs
            hadoop.hdfs.server.datanode.TestBlockRecovery
            hadoop.hdfs.security.TestDelegationToken
            hadoop.hdfs.server.namenode.TestBlockUnderConstruction
            hadoop.hdfs.TestFileConcurrentReader
            hadoop.hdfs.TestDFSClientRetries
            hadoop.hdfs.server.namenode.TestINodeAttributeProvider
            hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens
            hadoop.hdfs.TestParallelShortCircuitReadUnCached
            hadoop.hdfs.tools.TestDFSAdmin
            hadoop.hdfs.TestEncryptionZonesWithKMS
            hadoop.hdfs.crypto.TestHdfsCryptoStreams
            hadoop.hdfs.TestRead
            hadoop.hdfs.TestDFSInotifyEventInputStream
            hadoop.hdfs.server.datanode.TestDataNodeInitStorage
            hadoop.hdfs.server.namenode.TestFSNamesystemMBean
            hadoop.fs.loadGenerator.TestLoadGenerator
            hadoop.fs.contract.hdfs.TestHDFSContractAppend
            hadoop.fs.viewfs.TestViewFileSystemWithAcls
            hadoop.hdfs.web.TestWebHDFSXAttr
            hadoop.hdfs.server.datanode.TestIncrementalBlockReports
            hadoop.hdfs.web.TestWebHdfsTokens
            hadoop.hdfs.TestSetrepIncreasing
            hadoop.fs.TestFcHdfsSetUMask
            hadoop.hdfs.TestLargeBlock
            hadoop.hdfs.TestFileAppend3
            hadoop.hdfs.server.blockmanagement.TestSequentialBlockId
            hadoop.hdfs.TestDFSClientFailover
            hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotStatsMXBean
            hadoop.hdfs.server.datanode.TestReadOnlySharedStorage
            hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
            hadoop.hdfs.server.namenode.ha.TestHAMetrics
            hadoop.hdfs.server.namenode.TestFSDirectory
            hadoop.hdfs.server.namenode.TestFSImage
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy
            hadoop.hdfs.server.namenode.TestAddBlockRetry
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.TestPipelines
            hadoop.hdfs.TestSetrepDecreasing
            hadoop.hdfs.TestAppendDifferentChecksum
            hadoop.hdfs.TestSeekBug
            hadoop.hdfs.TestFileAppend4
            hadoop.hdfs.TestDFSRemove
            hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
            hadoop.hdfs.TestBlockReaderLocalLegacy
            hadoop.hdfs.TestBlocksScheduledCounter
            hadoop.fs.viewfs.TestViewFsFileStatusHdfs
            hadoop.hdfs.TestSnapshotCommands
            hadoop.fs.shell.TestHdfsTextCommand
            hadoop.hdfs.TestWriteConfigurationToDFS
            hadoop.hdfs.TestDFSRollback
            hadoop.hdfs.TestBlockStoragePolicy
            hadoop.hdfs.TestFSInputChecker
            hadoop.hdfs.TestParallelShortCircuitRead
            hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement
            hadoop.hdfs.TestAbandonBlock
            hadoop.fs.TestGlobPaths
            hadoop.hdfs.TestDataTransferKeepalive
            hadoop.hdfs.TestHDFSTrash
            hadoop.hdfs.server.namenode.TestNameNodeAcl
            hadoop.fs.contract.hdfs.TestHDFSContractRename
            hadoop.hdfs.TestDFSInputStream
            hadoop.hdfs.TestHDFSServerPorts
            hadoop.hdfs.TestRenameWhileOpen
            hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot
            hadoop.hdfs.TestEncryptedTransfer
            hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol
            hadoop.hdfs.TestHDFSFileSystemContract
            hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes
            hadoop.net.TestNetworkTopology
            hadoop.hdfs.security.token.block.TestBlockToken
            hadoop.hdfs.server.datanode.TestIncrementalBrVariations
            hadoop.hdfs.server.datanode.TestHSync
            hadoop.hdfs.server.namenode.TestNamenodeRetryCache
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
            hadoop.hdfs.TestDatanodeLayoutUpgrade
            hadoop.hdfs.TestWriteBlockGetsBlockLengthHint
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
            hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
            hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
            hadoop.hdfs.server.namenode.ha.TestDNFencing
            hadoop.hdfs.server.namenode.TestCreateEditsLog
            hadoop.hdfs.server.namenode.TestFileContextXAttr



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12738937/HDFS-8572.000.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a7a7768
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11306/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11306/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11306/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11306/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 0s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 40s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 11s The applied patch generated 2 new checkstyle issues (total was 154, now 147). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse. +1 findbugs 3m 16s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 12s Pre-build of native portion -1 hdfs tests 50m 53s Tests failed in hadoop-hdfs.     97m 26s   Reason Tests Failed unit tests hadoop.hdfs.TestFileStatus   hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes   hadoop.hdfs.server.namenode.TestINodeFile   hadoop.fs.contract.hdfs.TestHDFSContractOpen   hadoop.hdfs.server.datanode.TestFsDatasetCache   hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart   hadoop.hdfs.TestFileCreationDelete   hadoop.hdfs.server.namenode.ha.TestHASafeMode   hadoop.hdfs.TestEncryptionZonesWithHA   hadoop.fs.contract.hdfs.TestHDFSContractMkdir   hadoop.hdfs.TestDFSShell   hadoop.hdfs.server.namenode.TestNameNodeXAttr   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestDecommission   hadoop.hdfs.TestDatanodeDeath   hadoop.hdfs.server.namenode.TestFSEditLogLoader   hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages   hadoop.hdfs.server.datanode.TestCachingStrategy   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.cli.TestXAttrCLI   hadoop.hdfs.server.namenode.TestDeleteRace   hadoop.hdfs.server.namenode.TestParallelImageWrite   hadoop.hdfs.server.namenode.TestSaveNamespace   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.TestDFSRename   hadoop.hdfs.server.namenode.TestFsck   hadoop.hdfs.server.namenode.ha.TestHarFileSystemWithHA   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement   hadoop.hdfs.TestFileCreationEmpty   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration   hadoop.hdfs.server.datanode.TestDeleteBlockPool   hadoop.hdfs.TestRemoteBlockReader2   hadoop.hdfs.server.namenode.TestStorageRestore   hadoop.hdfs.server.namenode.TestFileLimit   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots   hadoop.hdfs.web.TestWebHdfsFileSystemContract   hadoop.hdfs.TestDFSFinalize   hadoop.hdfs.server.namenode.TestFileContextAcl   hadoop.hdfs.TestDatanodeRegistration   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.TestDataTransferProtocol   hadoop.fs.viewfs.TestViewFsWithXAttrs   hadoop.hdfs.server.datanode.TestDiskError   hadoop.hdfs.TestFsShellPermission   hadoop.hdfs.server.namenode.TestMalformedURLs   hadoop.hdfs.TestBalancerBandwidth   hadoop.hdfs.TestReadWhileWriting   hadoop.hdfs.TestDisableConnCache   hadoop.hdfs.TestIsMethodSupported   hadoop.hdfs.TestDFSClientExcludedNodes   hadoop.hdfs.TestParallelShortCircuitReadNoChecksum   hadoop.hdfs.TestFileCreationClient   hadoop.cli.TestAclCLI   hadoop.hdfs.TestFSOutputSummer   hadoop.fs.TestHDFSFileContextMainOperations   hadoop.hdfs.TestDFSStartupVersions   hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters   hadoop.hdfs.TestParallelShortCircuitLegacyRead   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.TestFileAppend2   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory   hadoop.hdfs.TestParallelUnixDomainRead   hadoop.hdfs.server.datanode.TestTransferRbw   hadoop.fs.contract.hdfs.TestHDFSContractSeek   hadoop.hdfs.TestLocalDFS   hadoop.hdfs.server.mover.TestStorageMover   hadoop.hdfs.TestGetFileChecksum   hadoop.hdfs.server.namenode.TestMetadataVersionOutput   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.server.namenode.TestFSImageWithAcl   hadoop.fs.TestUnbuffer   hadoop.hdfs.TestDFSPermission   hadoop.hdfs.TestParallelRead   hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks   hadoop.hdfs.server.namenode.TestAddBlock   hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles   hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold   hadoop.hdfs.server.namenode.TestMetaSave   hadoop.hdfs.TestSmallBlock   hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks   hadoop.hdfs.web.TestHttpsFileSystem   hadoop.tracing.TestTracing   hadoop.hdfs.TestListFilesInDFS   hadoop.hdfs.TestFileAppendRestart   hadoop.hdfs.TestBlockReaderLocal   hadoop.hdfs.server.datanode.TestTriggerBlockReport   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter   hadoop.fs.contract.hdfs.TestHDFSContractConcat   hadoop.hdfs.TestReplication   hadoop.hdfs.server.namenode.TestEditLog   hadoop.security.TestRefreshUserMappings   hadoop.hdfs.server.namenode.snapshot.TestFileContextSnapshot   hadoop.hdfs.server.mover.TestMover   hadoop.hdfs.TestSafeMode   hadoop.hdfs.server.namenode.TestHDFSConcat   hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS   hadoop.hdfs.server.namenode.TestFSImageWithXAttr   hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer   hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs   hadoop.security.TestPermissionSymlinks   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.TestHFlush   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.fs.TestSymlinkHdfsDisable   hadoop.hdfs.server.namenode.TestQuotaByStorageType   hadoop.hdfs.TestFileCreation   hadoop.hdfs.server.namenode.snapshot.TestDisallowModifyROSnapshot   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.server.namenode.ha.TestQuotasWithHA   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.namenode.ha.TestHAStateTransitions   hadoop.hdfs.server.namenode.snapshot.TestSetQuotaWithSnapshot   hadoop.hdfs.server.namenode.metrics.TestNNMetricFilesInGetListingOps   hadoop.hdfs.TestBlockReaderFactory   hadoop.hdfs.TestRemoteBlockReader   hadoop.hdfs.TestFetchImage   hadoop.hdfs.TestDFSOutputStream   hadoop.fs.TestUrlStreamHandler   hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot   hadoop.hdfs.TestModTime   hadoop.hdfs.TestLeaseRecovery2   hadoop.fs.viewfs.TestViewFileSystemWithXAttrs   hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.TestRestartDFS   hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade   hadoop.hdfs.TestInjectionForSimulatedStorage   hadoop.hdfs.TestMiniDFSCluster   hadoop.hdfs.server.datanode.TestFsDatasetCacheRevocation   hadoop.hdfs.server.namenode.snapshot.TestSnapshot   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.TestLease   hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.namenode.TestAuditLogs   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.TestDFSAddressConfig   hadoop.fs.TestSymlinkHdfsFileSystem   hadoop.hdfs.server.datanode.fsdataset.impl.TestRbwSpaceReservation   hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics   hadoop.tools.TestJMXGet   hadoop.hdfs.TestDFSMkdirs   hadoop.hdfs.TestWriteRead   hadoop.hdfs.TestClientBlockVerification   hadoop.fs.viewfs.TestViewFsDefaultValue   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles   hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages   hadoop.hdfs.TestDatanodeConfig   hadoop.hdfs.server.namenode.TestNameNodeResourceChecker   hadoop.hdfs.server.datanode.TestDataNodeECN   hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate   hadoop.hdfs.server.datanode.TestDataNodeMXBean   hadoop.hdfs.server.namenode.TestAclConfigFlag   hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd   hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA   hadoop.hdfs.server.namenode.TestBackupNode   hadoop.cli.TestHDFSCLI   hadoop.hdfs.server.namenode.ha.TestStandbyIsHot   hadoop.hdfs.server.namenode.TestAllowFormat   hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant   hadoop.hdfs.server.namenode.TestNameNodeRpcServer   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport   hadoop.fs.contract.hdfs.TestHDFSContractDelete   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery   hadoop.hdfs.server.namenode.TestCheckpoint   hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing   hadoop.hdfs.server.namenode.TestDeadDatanode   hadoop.hdfs.web.TestWebHdfsWithAuthenticationFilter   hadoop.hdfs.TestDatanodeReport   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.TestClientReportBadBlock   hadoop.hdfs.server.namenode.TestAuditLogger   hadoop.hdfs.server.datanode.TestDatanodeProtocolRetryPolicy   hadoop.hdfs.TestDFSShellGenericOptions   hadoop.security.TestPermission   hadoop.hdfs.web.TestWebHDFSAcl   hadoop.hdfs.TestMissingBlocksAlert   hadoop.fs.viewfs.TestViewFsHdfs   hadoop.hdfs.TestClose   hadoop.hdfs.tools.TestDebugAdmin   hadoop.hdfs.TestFileCorruption   hadoop.hdfs.server.namenode.TestFsckWithMultipleNameNodes   hadoop.fs.TestFcHdfsCreateMkdir   hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots   hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing   hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer   hadoop.hdfs.server.namenode.TestXAttrConfigFlag   hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication   hadoop.fs.permission.TestStickyBit   hadoop.fs.viewfs.TestViewFsWithAcls   hadoop.hdfs.tools.TestStoragePolicyCommands   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.server.namenode.TestStartup   hadoop.hdfs.TestMultiThreadedHflush   hadoop.TestGenericRefresh   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.TestQuota   hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestConnCache   hadoop.fs.viewfs.TestViewFsAtHdfsRoot   hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling   hadoop.hdfs.server.blockmanagement.TestPendingReplication   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.namenode.TestSecurityTokenEditLog   hadoop.tracing.TestTracingShortCircuitLocalRead   hadoop.fs.TestFcHdfsPermission   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy   hadoop.hdfs.TestLeaseRecovery   hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics   hadoop.hdfs.TestFileLengthOnClusterRestart   hadoop.fs.contract.hdfs.TestHDFSContractRootDirectory   hadoop.fs.contract.hdfs.TestHDFSContractCreate   hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN   hadoop.hdfs.TestFileAppend   hadoop.hdfs.web.TestWebHDFS   hadoop.hdfs.TestGetBlocks   hadoop.hdfs.server.namenode.TestProcessCorruptBlocks   hadoop.fs.TestResolveHdfsSymlink   hadoop.hdfs.server.datanode.TestStorageReport   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl   hadoop.hdfs.TestDFSUpgrade   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.TestPread   hadoop.hdfs.server.namenode.TestCacheDirectives   hadoop.hdfs.server.namenode.TestNameNodeMXBean   hadoop.hdfs.server.namenode.TestLargeDirectoryDelete   hadoop.hdfs.server.namenode.TestFSImageWithSnapshot   hadoop.tracing.TestTraceAdmin   hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap   hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer   hadoop.hdfs.server.namenode.snapshot.TestSnapshotRename   hadoop.hdfs.server.namenode.TestNameEditsConfigs   hadoop.cli.TestCacheAdminCLI   hadoop.hdfs.server.datanode.TestDataNodeExit   hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage   hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication   hadoop.hdfs.server.datanode.TestRefreshNamenodes   hadoop.hdfs.web.TestWebHDFSForHA   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport   hadoop.hdfs.server.namenode.TestHostsFiles   hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots   hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength   hadoop.hdfs.web.TestFSMainOperationsWebHdfs   hadoop.hdfs.server.namenode.TestEditLogRace   hadoop.hdfs.TestReservedRawPaths   hadoop.TestRefreshCallQueue   hadoop.hdfs.TestSetTimes   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica   hadoop.hdfs.server.namenode.ha.TestFailoverWithBlockTokensEnabled   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.fs.TestEnhancedByteBufferAccess   hadoop.cli.TestCryptoAdminCLI   hadoop.hdfs.TestClientProtocolForPipelineRecovery   hadoop.hdfs.TestDFSStorageStateRecovery   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.namenode.TestDefaultBlockPlacementPolicy   hadoop.hdfs.TestListFilesInFileContext   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.TestBlockMissingException   hadoop.hdfs.server.namenode.TestSnapshotPathINodes   hadoop.hdfs.server.datanode.web.dtp.TestDtpHttp2   hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade   hadoop.fs.viewfs.TestViewFileSystemHdfs   hadoop.hdfs.server.datanode.TestBlockRecovery   hadoop.hdfs.security.TestDelegationToken   hadoop.hdfs.server.namenode.TestBlockUnderConstruction   hadoop.hdfs.TestFileConcurrentReader   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.namenode.TestINodeAttributeProvider   hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens   hadoop.hdfs.TestParallelShortCircuitReadUnCached   hadoop.hdfs.tools.TestDFSAdmin   hadoop.hdfs.TestEncryptionZonesWithKMS   hadoop.hdfs.crypto.TestHdfsCryptoStreams   hadoop.hdfs.TestRead   hadoop.hdfs.TestDFSInotifyEventInputStream   hadoop.hdfs.server.datanode.TestDataNodeInitStorage   hadoop.hdfs.server.namenode.TestFSNamesystemMBean   hadoop.fs.loadGenerator.TestLoadGenerator   hadoop.fs.contract.hdfs.TestHDFSContractAppend   hadoop.fs.viewfs.TestViewFileSystemWithAcls   hadoop.hdfs.web.TestWebHDFSXAttr   hadoop.hdfs.server.datanode.TestIncrementalBlockReports   hadoop.hdfs.web.TestWebHdfsTokens   hadoop.hdfs.TestSetrepIncreasing   hadoop.fs.TestFcHdfsSetUMask   hadoop.hdfs.TestLargeBlock   hadoop.hdfs.TestFileAppend3   hadoop.hdfs.server.blockmanagement.TestSequentialBlockId   hadoop.hdfs.TestDFSClientFailover   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.server.namenode.snapshot.TestSnapshotStatsMXBean   hadoop.hdfs.server.datanode.TestReadOnlySharedStorage   hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer   hadoop.hdfs.server.namenode.ha.TestHAMetrics   hadoop.hdfs.server.namenode.TestFSDirectory   hadoop.hdfs.server.namenode.TestFSImage   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy   hadoop.hdfs.server.namenode.TestAddBlockRetry   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.TestPipelines   hadoop.hdfs.TestSetrepDecreasing   hadoop.hdfs.TestAppendDifferentChecksum   hadoop.hdfs.TestSeekBug   hadoop.hdfs.TestFileAppend4   hadoop.hdfs.TestDFSRemove   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer   hadoop.hdfs.TestBlockReaderLocalLegacy   hadoop.hdfs.TestBlocksScheduledCounter   hadoop.fs.viewfs.TestViewFsFileStatusHdfs   hadoop.hdfs.TestSnapshotCommands   hadoop.fs.shell.TestHdfsTextCommand   hadoop.hdfs.TestWriteConfigurationToDFS   hadoop.hdfs.TestDFSRollback   hadoop.hdfs.TestBlockStoragePolicy   hadoop.hdfs.TestFSInputChecker   hadoop.hdfs.TestParallelShortCircuitRead   hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement   hadoop.hdfs.TestAbandonBlock   hadoop.fs.TestGlobPaths   hadoop.hdfs.TestDataTransferKeepalive   hadoop.hdfs.TestHDFSTrash   hadoop.hdfs.server.namenode.TestNameNodeAcl   hadoop.fs.contract.hdfs.TestHDFSContractRename   hadoop.hdfs.TestDFSInputStream   hadoop.hdfs.TestHDFSServerPorts   hadoop.hdfs.TestRenameWhileOpen   hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot   hadoop.hdfs.TestEncryptedTransfer   hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol   hadoop.hdfs.TestHDFSFileSystemContract   hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes   hadoop.net.TestNetworkTopology   hadoop.hdfs.security.token.block.TestBlockToken   hadoop.hdfs.server.datanode.TestIncrementalBrVariations   hadoop.hdfs.server.datanode.TestHSync   hadoop.hdfs.server.namenode.TestNamenodeRetryCache   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.hdfs.TestDatanodeLayoutUpgrade   hadoop.hdfs.TestWriteBlockGetsBlockLengthHint   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup   hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality   hadoop.hdfs.server.namenode.TestListCorruptFileBlocks   hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.server.namenode.TestCreateEditsLog   hadoop.hdfs.server.namenode.TestFileContextXAttr Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12738937/HDFS-8572.000.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a7a7768 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11306/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11306/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11306/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11306/console This message was automatically generated.
          Hide
          yzhangal Yongjun Zhang added a comment -

          HI Haohui Mai,

          Thanks for reporting the issue and the patch. Coincidentally I am investigating this issue this couple of days and identified that HDFS-7279 caused the problem.

          I briefly looked at the patch, found that the test failure of your patch is because it always set the cluster to secure in the following code:

           HttpServer2.Builder builder = new HttpServer2.Builder()
                  .setName("datanode")
                  .setConf(confForInfoServer)
                  .setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
                  .setSecurityEnabled(true) <============================================
                  .setUsernameConfKey(DFS_WEB_AUTHENTICATION_KERBEROS_PRINCIPAL_KEY)
                  .setKeytabConfKey(DFS_WEB_AUTHENTICATION_KERBEROS_KEYTAB_KEY)
                  .hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
                  .addEndpoint(URI.create("http://localhost:0"))
                  .setFindPort(true);
          

          Suggest to change it to

          boolean isSecure = UserGroupInformation.isSecurityEnabled();
                  .setSecurityEnabled(isSecure)
          

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - HI Haohui Mai , Thanks for reporting the issue and the patch. Coincidentally I am investigating this issue this couple of days and identified that HDFS-7279 caused the problem. I briefly looked at the patch, found that the test failure of your patch is because it always set the cluster to secure in the following code: HttpServer2.Builder builder = new HttpServer2.Builder() .setName( "datanode" ) .setConf(confForInfoServer) .setACL( new AccessControlList(conf.get(DFS_ADMIN, " " ))) .setSecurityEnabled( true ) <============================================ .setUsernameConfKey(DFS_WEB_AUTHENTICATION_KERBEROS_PRINCIPAL_KEY) .setKeytabConfKey(DFS_WEB_AUTHENTICATION_KERBEROS_KEYTAB_KEY) .hostName(getHostnameForSpnegoPrincipal(confForInfoServer)) .addEndpoint(URI.create( "http: //localhost:0" )) .setFindPort( true ); Suggest to change it to boolean isSecure = UserGroupInformation.isSecurityEnabled(); .setSecurityEnabled(isSecure) Thanks.
          Hide
          yzhangal Yongjun Zhang added a comment -

          One question

            private static String getHostnameForSpnegoPrincipal(Configuration conf) {
              String addr = conf.getTrimmed(DFS_DATANODE_HTTP_ADDRESS_KEY, null);
              if (addr == null) {
                addr = conf.getTrimmed(DFS_DATANODE_HTTPS_ADDRESS_KEY,
                                       DFS_DATANODE_HTTPS_ADDRESS_DEFAULT);
              }
              InetSocketAddress inetSocker = NetUtils.createSocketAddr(addr);
              String hostName = inetSocker.getHostString();
              return hostName;
            }
          

          Shouldn't this code check whether it's secure cluster or not then read the corresponding config property, instead of checking "==nul" with one then try the other?

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - One question private static String getHostnameForSpnegoPrincipal(Configuration conf) { String addr = conf.getTrimmed(DFS_DATANODE_HTTP_ADDRESS_KEY, null ); if (addr == null ) { addr = conf.getTrimmed(DFS_DATANODE_HTTPS_ADDRESS_KEY, DFS_DATANODE_HTTPS_ADDRESS_DEFAULT); } InetSocketAddress inetSocker = NetUtils.createSocketAddr(addr); String hostName = inetSocker.getHostString(); return hostName; } Shouldn't this code check whether it's secure cluster or not then read the corresponding config property, instead of checking "==nul" with one then try the other? Thanks.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Haohui Mai,

          Thanks for the new rev. Two comments here:

          1. I noticed that your new rev removed the call to

                  .setSecurityEnabled(...)
                  .setUsernameConfKey(DFS_WEB_AUTHENTICATION_KERBEROS_PRINCIPAL_KEY)
                  .setKeytabConfKey(DFS_WEB_AUTHENTICATION_KERBEROS_KEYTAB_KEY)
          

          So the datanode http server's securityEnabled is the default value false and the following code is not executed:

             if (this.securityEnabled) {
                server.initSpnego(conf, hostName, usernameConfKey, keytabConfKey);
             }
          

          Is this expected? if so, why we don't need to set securityEnabled here? when do we need to set it?

          2. There is a redundant line to be removed:

          import static org.apache.hadoop.hdfs.DFSConfigKeys.*;
          

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Hi Haohui Mai , Thanks for the new rev. Two comments here: 1. I noticed that your new rev removed the call to .setSecurityEnabled(...) .setUsernameConfKey(DFS_WEB_AUTHENTICATION_KERBEROS_PRINCIPAL_KEY) .setKeytabConfKey(DFS_WEB_AUTHENTICATION_KERBEROS_KEYTAB_KEY) So the datanode http server's securityEnabled is the default value false and the following code is not executed: if ( this .securityEnabled) { server.initSpnego(conf, hostName, usernameConfKey, keytabConfKey); } Is this expected? if so, why we don't need to set securityEnabled here? when do we need to set it? 2. There is a redundant line to be removed: import static org.apache.hadoop.hdfs.DFSConfigKeys.*; Thanks.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 13s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 48s There were no new javac warning messages.
          +1 javadoc 10m 6s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 17s The applied patch generated 2 new checkstyle issues (total was 154, now 147).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 17s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 23s Pre-build of native portion
          +1 hdfs tests 162m 13s Tests passed in hadoop-hdfs.
              209m 53s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12738989/HDFS-8572.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a7a7768
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11308/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11308/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11308/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11308/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 13s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 48s There were no new javac warning messages. +1 javadoc 10m 6s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 17s The applied patch generated 2 new checkstyle issues (total was 154, now 147). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 17s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 23s Pre-build of native portion +1 hdfs tests 162m 13s Tests passed in hadoop-hdfs.     209m 53s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12738989/HDFS-8572.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a7a7768 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11308/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11308/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11308/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11308/console This message was automatically generated.
          Hide
          wheat9 Haohui Mai added a comment -

          Is this expected? if so, why we don't need to set securityEnabled here? when do we need to set it?

          This is expected. DN is not supposed to support SPNEGO authentication at all. This patch is to quickly restore the old behavior before HDFS-7279 which will unblock the release. I defer the correct fix in a separate jira.

          Show
          wheat9 Haohui Mai added a comment - Is this expected? if so, why we don't need to set securityEnabled here? when do we need to set it? This is expected. DN is not supposed to support SPNEGO authentication at all. This patch is to quickly restore the old behavior before HDFS-7279 which will unblock the release. I defer the correct fix in a separate jira.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 51s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 29s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 12s The applied patch generated 2 new checkstyle issues (total was 154, now 147).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 14s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 12s Pre-build of native portion
          +1 hdfs tests 161m 6s Tests passed in hadoop-hdfs.
              207m 17s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12739011/HDFS-8572.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 95c73d4
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11310/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11310/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11310/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11310/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 51s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 29s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 12s The applied patch generated 2 new checkstyle issues (total was 154, now 147). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 14s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 12s Pre-build of native portion +1 hdfs tests 161m 6s Tests passed in hadoop-hdfs.     207m 17s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739011/HDFS-8572.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 95c73d4 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11310/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11310/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11310/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11310/console This message was automatically generated.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Haohui Mai,

          Thanks for the new rev to address my comment.

          Is this expected? if so, why we don't need to set securityEnabled here? when do we need to set it?
          This is expected. DN is not supposed to support SPNEGO authentication at all. This patch is to quickly restore the old behavior before HDFS-7279 which will unblock the release. I defer the correct fix in a separate jira.

          I can see that your change restored old behavior and I agree that we can defer any other fixes to separate jira. So I agree with rev 002 solution here. Two comments:

          • Based on your explanation, the jira title here may not be accurate. Would you please update it if so?
          • Would you please create new jira with some description of the correct fix you referred to?

          BTW, we have a similar case that we hope can get a fix and defer other issues to separate jiras. Aaron T. Myers had a long standing comment here
          https://issues.apache.org/jira/browse/HDFS-7037?focusedCommentId=14484301&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14484301
          waiting for you to reply. We keep having use issues without the fix even as of yesterdaty. Would you please kindly revisit and reply to Aaron T. Myers's comment?

          Thanks much.

          Show
          yzhangal Yongjun Zhang added a comment - Hi Haohui Mai , Thanks for the new rev to address my comment. Is this expected? if so, why we don't need to set securityEnabled here? when do we need to set it? This is expected. DN is not supposed to support SPNEGO authentication at all. This patch is to quickly restore the old behavior before HDFS-7279 which will unblock the release. I defer the correct fix in a separate jira. I can see that your change restored old behavior and I agree that we can defer any other fixes to separate jira. So I agree with rev 002 solution here. Two comments: Based on your explanation, the jira title here may not be accurate. Would you please update it if so? Would you please create new jira with some description of the correct fix you referred to? BTW, we have a similar case that we hope can get a fix and defer other issues to separate jiras. Aaron T. Myers had a long standing comment here https://issues.apache.org/jira/browse/HDFS-7037?focusedCommentId=14484301&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14484301 waiting for you to reply. We keep having use issues without the fix even as of yesterdaty. Would you please kindly revisit and reply to Aaron T. Myers 's comment? Thanks much.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Haohui Mai,

          I both checked that the patch restored the old behavior, and tested in a real cluster. Since now that you have addressed my review comments. I'm +1 on 002 pending your addressing my questions in my last comment.

          Thanks much.

          Show
          yzhangal Yongjun Zhang added a comment - Hi Haohui Mai , I both checked that the patch restored the old behavior, and tested in a real cluster. Since now that you have addressed my review comments. I'm +1 on 002 pending your addressing my questions in my last comment. Thanks much.
          Hide
          wheat9 Haohui Mai added a comment -

          I've committed the patch to trunk and branch-2. Thanks Yongjun for the reviews.

          Show
          wheat9 Haohui Mai added a comment - I've committed the patch to trunk and branch-2. Thanks Yongjun for the reviews.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Haohui,

          Thanks again for the solution. As we discussed, would you please create a follow-up jira? Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Hi Haohui, Thanks again for the solution. As we discussed, would you please create a follow-up jira? Thanks.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8013 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8013/)
          HDFS-8572. DN always uses HTTP/localhost@REALM principals in SPNEGO. Contributed by Haohui Mai. (wheat9: rev eef7b50e23f9960e4bb61d9db6754a2300bc06eb)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8013 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8013/ ) HDFS-8572 . DN always uses HTTP/localhost@REALM principals in SPNEGO. Contributed by Haohui Mai. (wheat9: rev eef7b50e23f9960e4bb61d9db6754a2300bc06eb) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #227 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/227/)
          HDFS-8572. DN always uses HTTP/localhost@REALM principals in SPNEGO. Contributed by Haohui Mai. (wheat9: rev eef7b50e23f9960e4bb61d9db6754a2300bc06eb)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #227 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/227/ ) HDFS-8572 . DN always uses HTTP/localhost@REALM principals in SPNEGO. Contributed by Haohui Mai. (wheat9: rev eef7b50e23f9960e4bb61d9db6754a2300bc06eb) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #957 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/957/)
          HDFS-8572. DN always uses HTTP/localhost@REALM principals in SPNEGO. Contributed by Haohui Mai. (wheat9: rev eef7b50e23f9960e4bb61d9db6754a2300bc06eb)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #957 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/957/ ) HDFS-8572 . DN always uses HTTP/localhost@REALM principals in SPNEGO. Contributed by Haohui Mai. (wheat9: rev eef7b50e23f9960e4bb61d9db6754a2300bc06eb) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2155 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2155/)
          HDFS-8572. DN always uses HTTP/localhost@REALM principals in SPNEGO. Contributed by Haohui Mai. (wheat9: rev eef7b50e23f9960e4bb61d9db6754a2300bc06eb)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2155 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2155/ ) HDFS-8572 . DN always uses HTTP/localhost@REALM principals in SPNEGO. Contributed by Haohui Mai. (wheat9: rev eef7b50e23f9960e4bb61d9db6754a2300bc06eb) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #216 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/216/)
          HDFS-8572. DN always uses HTTP/localhost@REALM principals in SPNEGO. Contributed by Haohui Mai. (wheat9: rev eef7b50e23f9960e4bb61d9db6754a2300bc06eb)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #216 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/216/ ) HDFS-8572 . DN always uses HTTP/localhost@REALM principals in SPNEGO. Contributed by Haohui Mai. (wheat9: rev eef7b50e23f9960e4bb61d9db6754a2300bc06eb) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #225 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/225/)
          HDFS-8572. DN always uses HTTP/localhost@REALM principals in SPNEGO. Contributed by Haohui Mai. (wheat9: rev eef7b50e23f9960e4bb61d9db6754a2300bc06eb)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #225 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/225/ ) HDFS-8572 . DN always uses HTTP/localhost@REALM principals in SPNEGO. Contributed by Haohui Mai. (wheat9: rev eef7b50e23f9960e4bb61d9db6754a2300bc06eb) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2173 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2173/)
          HDFS-8572. DN always uses HTTP/localhost@REALM principals in SPNEGO. Contributed by Haohui Mai. (wheat9: rev eef7b50e23f9960e4bb61d9db6754a2300bc06eb)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2173 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2173/ ) HDFS-8572 . DN always uses HTTP/localhost@REALM principals in SPNEGO. Contributed by Haohui Mai. (wheat9: rev eef7b50e23f9960e4bb61d9db6754a2300bc06eb) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          HI Haohui Mai

          while committing this patch,do you remove following purposefully in branch-2..?

          -    this.infoServer.addInternalServlet(null, "/streamFile/*", StreamFile.class);
          -    this.infoServer.addInternalServlet(null, "/getFileChecksum/*",
          -        FileChecksumServlets.GetServlet.class);
          

          I think ,distcp will fail when we use hftp since it expect streamFile and throw following error..am I correct..? if you agree with me,would like raise separate jira and add back...

          Caused by: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.io.IOException: HTTP_OK expected, received 404
          at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.getInputStream(RetriableFileCopyCommand.java:302)
          at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:247)
          at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:183)
          at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:123)
          at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
          at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
          ... 11 more
          Caused by: java.io.IOException: HTTP_OK expected, received 404
          at org.apache.hadoop.hdfs.web.HftpFileSystem$RangeHeaderUrlOpener.connect(HftpFileSystem.java:367)
          at org.apache.hadoop.hdfs.web.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:133)
          at org.apache.hadoop.hdfs.web.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:115)
          at org.apache.hadoop.hdfs.web.ByteRangeInputStream.<init>(ByteRangeInputStream.java:100)
          at org.apache.hadoop.hdfs.web.HftpFileSystem$RangeHeaderInputStream.<init>(HftpFileSystem.java:376)
          at org.apache.hadoop.hdfs.web.HftpFileSystem$RangeHeaderInputStream.<init>(HftpFileSystem.java:381)
          at org.apache.hadoop.hdfs.web.HftpFileSystem.open(HftpFileSystem.java:397)
          at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
          
          Show
          brahmareddy Brahma Reddy Battula added a comment - HI Haohui Mai while committing this patch,do you remove following purposefully in branch-2..? - this .infoServer.addInternalServlet( null , "/streamFile/*" , StreamFile.class); - this .infoServer.addInternalServlet( null , "/getFileChecksum/*" , - FileChecksumServlets.GetServlet.class); I think ,distcp will fail when we use hftp since it expect streamFile and throw following error..am I correct..? if you agree with me,would like raise separate jira and add back... Caused by: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.io.IOException: HTTP_OK expected, received 404 at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.getInputStream(RetriableFileCopyCommand.java:302) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:247) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:183) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:123) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) ... 11 more Caused by: java.io.IOException: HTTP_OK expected, received 404 at org.apache.hadoop.hdfs.web.HftpFileSystem$RangeHeaderUrlOpener.connect(HftpFileSystem.java:367) at org.apache.hadoop.hdfs.web.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:133) at org.apache.hadoop.hdfs.web.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:115) at org.apache.hadoop.hdfs.web.ByteRangeInputStream.<init>(ByteRangeInputStream.java:100) at org.apache.hadoop.hdfs.web.HftpFileSystem$RangeHeaderInputStream.<init>(HftpFileSystem.java:376) at org.apache.hadoop.hdfs.web.HftpFileSystem$RangeHeaderInputStream.<init>(HftpFileSystem.java:381) at org.apache.hadoop.hdfs.web.HftpFileSystem.open(HftpFileSystem.java:397) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Haohui Mai can I raise separate jira to track this..?

          Show
          brahmareddy Brahma Reddy Battula added a comment - Haohui Mai can I raise separate jira to track this..?

            People

            • Assignee:
              wheat9 Haohui Mai
              Reporter:
              wheat9 Haohui Mai
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development