Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11984

Enable parallel JUnit tests in pre-commit.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: build, scripts, test
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      HADOOP-9287 and related issues implemented the parallel-tests Maven profile for running JUnit tests in multiple concurrent processes. This issue proposes to activate that profile during pre-commit to speed up execution.

      1. HADOOP-11984.001.patch
        2 kB
        Chris Nauroth
      2. HADOOP-11984.002.patch
        2 kB
        Chris Nauroth
      3. HADOOP-11984.003.patch
        7 kB
        Chris Nauroth
      4. HADOOP-11984.004.patch
        5 kB
        Chris Nauroth
      5. HADOOP-11984.005.patch
        6 kB
        Chris Nauroth
      6. HADOOP-11984.006.patch
        8 kB
        Chris Nauroth
      7. HADOOP-11984.007.patch
        23 kB
        Chris Nauroth
      8. HADOOP-11984.008.patch
        24 kB
        Chris Nauroth
      9. HADOOP-11984.009.patch
        14 kB
        Chris Nauroth
      10. HADOOP-11984.010.patch
        15 kB
        Chris Nauroth
      11. HADOOP-11984.011.patch
        15 kB
        Chris Nauroth
      12. HADOOP-11984.013.patch
        25 kB
        Chris Nauroth
      13. HADOOP-11984.014.patch
        26 kB
        Chris Nauroth
      14. HADOOP-11984.015.patch
        26 kB
        Chris Nauroth
      15. HADOOP-11984.016.patch
        26 kB
        Chris Nauroth
      16. HADOOP-11984.017.patch
        26 kB
        Chris Nauroth
      17. HADOOP-11984.019.patch
        27 kB
        Vinayakumar B
      18. HADOOP-11984.020.patch
        33 kB
        Vinayakumar B
      19. HADOOP-11984.021.patch
        36 kB
        Vinayakumar B
      20. HADOOP-11984.022.patch
        20 kB
        Vinayakumar B
      21. HADOOP-11984.023.patch
        19 kB
        Vinayakumar B
      22. HADOOP-11984.024.patch
        19 kB
        Chris Nauroth
      23. HADOOP-11984.HADOOP-12111.018.patch
        24 kB
        Allen Wittenauer

        Issue Links

          Activity

          Hide
          cnauroth Chris Nauroth added a comment -

          This patch adds the -Pparallel-tests flag to test-patch.sh. I also included a bogus test class in hadoop-hdfs, just to fake it into triggering those tests. Let's see how we do.

          Show
          cnauroth Chris Nauroth added a comment - This patch adds the -Pparallel-tests flag to test-patch.sh. I also included a bogus test class in hadoop-hdfs, just to fake it into triggering those tests. Let's see how we do.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to test-patch or smart-apply-patch has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6709/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6709/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 5m 13s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 34s There were no new javac warning messages.
          +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 13s There were no new checkstyle issues.
          -1 shellcheck 0m 8s The applied patch generated 13 new shellcheck (v0.3.3) issues (total was 38, now 51).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
          +1 findbugs 3m 3s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 native 1m 29s Pre-build of native portion
          -1 hdfs tests 42m 30s Tests failed in hadoop-hdfs.
              64m 35s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          Timed out tests org.apache.hadoop.fs.TestUrlStreamHandlerFactory
            org.apache.hadoop.fs.TestResolveHdfsSymlink
            org.apache.hadoop.fs.TestFcHdfsCreateMkdir
            org.apache.hadoop.fs.contract.hdfs.TestHDFSContractOpen
            org.apache.hadoop.fs.contract.hdfs.TestHDFSContractCreate
            org.apache.hadoop.fs.contract.hdfs.TestHDFSContractDelete
            org.apache.hadoop.fs.viewfs.TestViewFileSystemWithXAttrs
            org.apache.hadoop.fs.TestFcHdfsPermission
            org.apache.hadoop.fs.shell.TestHdfsTextCommand
            org.apache.hadoop.fs.TestEnhancedByteBufferAccess
            org.apache.hadoop.hdfs.TestFileCreationEmpty
            org.apache.hadoop.hdfs.TestAppendSnapshotTruncate
            org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            org.apache.hadoop.hdfs.TestDFSClientRetries
            org.apache.hadoop.hdfs.web.TestWebHdfsTimeouts
            org.apache.hadoop.hdfs.util.TestDiff
            org.apache.hadoop.hdfs.util.TestByteArrayManager
            org.apache.hadoop.hdfs.server.balancer.TestBalancer
            org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
            org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete
            org.apache.hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots
            org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks
            org.apache.hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
            org.apache.hadoop.hdfs.server.namenode.TestINodeAttributeProvider
            org.apache.hadoop.hdfs.server.namenode.TestAclTransformation
            org.apache.hadoop.hdfs.server.namenode.TestFSImageStorageInspector
            org.apache.hadoop.hdfs.server.namenode.TestEditLogJournalFailures
            org.apache.hadoop.hdfs.server.namenode.TestDeleteRace
            org.apache.hadoop.hdfs.server.namenode.TestSecondaryWebUi
            org.apache.hadoop.hdfs.server.namenode.TestEditLog
            org.apache.hadoop.hdfs.server.namenode.TestINodeFile
            org.apache.hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
            org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
            org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup
            org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager
            org.apache.hadoop.hdfs.TestFSInputChecker
            org.apache.hadoop.hdfs.TestDFSRename
            org.apache.hadoop.hdfs.protocol.TestExtendedBlock
            org.apache.hadoop.hdfs.TestAbandonBlock
            org.apache.hadoop.hdfs.TestMissingBlocksAlert
            org.apache.hadoop.hdfs.TestDatanodeLayoutUpgrade
            org.apache.hadoop.hdfs.TestBlockReaderLocal
            org.apache.hadoop.hdfs.TestMiniDFSCluster
            org.apache.hadoop.hdfs.TestSeekBug
            org.apache.hadoop.hdfs.crypto.TestHdfsCryptoStreams
            org.apache.hadoop.hdfs.TestDFSAddressConfig
            org.apache.hadoop.hdfs.TestPread
            org.apache.hadoop.hdfs.tools.TestDebugAdmin
            org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA
            org.apache.hadoop.TestGenericRefresh
            org.apache.hadoop.cli.TestHDFSCLI
            org.apache.hadoop.tools.TestHdfsConfigFields



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12733306/HADOOP-11984.001.patch
          Optional Tests shellcheck javac unit findbugs checkstyle
          git revision trunk / b0ad644
          shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/6709/artifact/patchprocess/diffpatchshellcheck.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6709/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6709/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6709/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 5m 13s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 34s There were no new javac warning messages. +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 13s There were no new checkstyle issues. -1 shellcheck 0m 8s The applied patch generated 13 new shellcheck (v0.3.3) issues (total was 38, now 51). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 findbugs 3m 3s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 1m 29s Pre-build of native portion -1 hdfs tests 42m 30s Tests failed in hadoop-hdfs.     64m 35s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots Timed out tests org.apache.hadoop.fs.TestUrlStreamHandlerFactory   org.apache.hadoop.fs.TestResolveHdfsSymlink   org.apache.hadoop.fs.TestFcHdfsCreateMkdir   org.apache.hadoop.fs.contract.hdfs.TestHDFSContractOpen   org.apache.hadoop.fs.contract.hdfs.TestHDFSContractCreate   org.apache.hadoop.fs.contract.hdfs.TestHDFSContractDelete   org.apache.hadoop.fs.viewfs.TestViewFileSystemWithXAttrs   org.apache.hadoop.fs.TestFcHdfsPermission   org.apache.hadoop.fs.shell.TestHdfsTextCommand   org.apache.hadoop.fs.TestEnhancedByteBufferAccess   org.apache.hadoop.hdfs.TestFileCreationEmpty   org.apache.hadoop.hdfs.TestAppendSnapshotTruncate   org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   org.apache.hadoop.hdfs.TestDFSClientRetries   org.apache.hadoop.hdfs.web.TestWebHdfsTimeouts   org.apache.hadoop.hdfs.util.TestDiff   org.apache.hadoop.hdfs.util.TestByteArrayManager   org.apache.hadoop.hdfs.server.balancer.TestBalancer   org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits   org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete   org.apache.hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots   org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks   org.apache.hadoop.hdfs.server.namenode.TestSecurityTokenEditLog   org.apache.hadoop.hdfs.server.namenode.TestINodeAttributeProvider   org.apache.hadoop.hdfs.server.namenode.TestAclTransformation   org.apache.hadoop.hdfs.server.namenode.TestFSImageStorageInspector   org.apache.hadoop.hdfs.server.namenode.TestEditLogJournalFailures   org.apache.hadoop.hdfs.server.namenode.TestDeleteRace   org.apache.hadoop.hdfs.server.namenode.TestSecondaryWebUi   org.apache.hadoop.hdfs.server.namenode.TestEditLog   org.apache.hadoop.hdfs.server.namenode.TestINodeFile   org.apache.hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage   org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup   org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager   org.apache.hadoop.hdfs.TestFSInputChecker   org.apache.hadoop.hdfs.TestDFSRename   org.apache.hadoop.hdfs.protocol.TestExtendedBlock   org.apache.hadoop.hdfs.TestAbandonBlock   org.apache.hadoop.hdfs.TestMissingBlocksAlert   org.apache.hadoop.hdfs.TestDatanodeLayoutUpgrade   org.apache.hadoop.hdfs.TestBlockReaderLocal   org.apache.hadoop.hdfs.TestMiniDFSCluster   org.apache.hadoop.hdfs.TestSeekBug   org.apache.hadoop.hdfs.crypto.TestHdfsCryptoStreams   org.apache.hadoop.hdfs.TestDFSAddressConfig   org.apache.hadoop.hdfs.TestPread   org.apache.hadoop.hdfs.tools.TestDebugAdmin   org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA   org.apache.hadoop.TestGenericRefresh   org.apache.hadoop.cli.TestHDFSCLI   org.apache.hadoop.tools.TestHdfsConfigFields Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733306/HADOOP-11984.001.patch Optional Tests shellcheck javac unit findbugs checkstyle git revision trunk / b0ad644 shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/6709/artifact/patchprocess/diffpatchshellcheck.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6709/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6709/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6709/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          That was pretty good overall. I see recent hadoop-hdfs test runs have been taking ~2.5 hours. This one was ~45 minutes. There are a couple of things to follow up on before we can commit this:

          1. TestRenameWithSnapshots failed due to OutOfMemoryError. I don't know yet if this is a problem in the test code or if we really over-taxed the Jenkins host by running too many JVM processes.
          2. Numerous tests were reported as timed out, but they actually passed. The test-patch.sh logic that checks for timed out tests works by looking for a "Running" line that is not followed by a "Tests run" line. This logic won't work with parallel execution, because the output from multiple JUnit processes gets interleaved non-deterministically. We'll need to figure out a different way to do the timed out check.
          3. The parallel-tests profile is only implemented for hadoop-common and hadoop-hdfs, so other sub-modules won't get the benefit yet. MapReduce is tracked in MAPREDUCE-4980. I don't think we have a YARN jira. We don't have to wait for these to get done. Those modules just won't get the benefit until they're done.

          Next, I'm going to try another patch with a forced hadoop-common change to see how that does.

          Show
          cnauroth Chris Nauroth added a comment - That was pretty good overall. I see recent hadoop-hdfs test runs have been taking ~2.5 hours. This one was ~45 minutes. There are a couple of things to follow up on before we can commit this: TestRenameWithSnapshots failed due to OutOfMemoryError . I don't know yet if this is a problem in the test code or if we really over-taxed the Jenkins host by running too many JVM processes. Numerous tests were reported as timed out, but they actually passed. The test-patch.sh logic that checks for timed out tests works by looking for a "Running" line that is not followed by a "Tests run" line. This logic won't work with parallel execution, because the output from multiple JUnit processes gets interleaved non-deterministically. We'll need to figure out a different way to do the timed out check. The parallel-tests profile is only implemented for hadoop-common and hadoop-hdfs, so other sub-modules won't get the benefit yet. MapReduce is tracked in MAPREDUCE-4980 . I don't think we have a YARN jira. We don't have to wait for these to get done. Those modules just won't get the benefit until they're done. Next, I'm going to try another patch with a forced hadoop-common change to see how that does.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to test-patch or smart-apply-patch has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6714/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6714/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 5m 11s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 31s There were no new javac warning messages.
          +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 3s There were no new checkstyle issues.
          -1 shellcheck 0m 8s The applied patch generated 13 new shellcheck (v0.3.3) issues (total was 38, now 51).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 37s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          -1 common tests 20m 33s Tests failed in hadoop-common.
              38m 29s  



          Reason Tests
          Failed unit tests hadoop.fs.contract.localfs.TestLocalFSContractOpen
            hadoop.fs.contract.localfs.TestLocalFSContractRename
            hadoop.fs.contract.localfs.TestLocalFSContractMkdir
            hadoop.ha.TestZKFailoverControllerStress
            hadoop.fs.contract.localfs.TestLocalFSContractDelete
            hadoop.fs.contract.localfs.TestLocalFSContractCreate
          Timed out tests org.apache.hadoop.service.TestServiceLifecycle
            org.apache.hadoop.service.TestCompositeService
            org.apache.hadoop.service.TestGlobalStateChangeListener
            org.apache.hadoop.ha.TestActiveStandbyElectorRealZK
            org.apache.hadoop.metrics.TestMetricsServlet
            org.apache.hadoop.metrics2.impl.TestMetricsVisitor
            org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
            org.apache.hadoop.metrics2.impl.TestGangliaMetrics
            org.apache.hadoop.metrics2.lib.TestMetricsRegistry
            org.apache.hadoop.metrics2.lib.TestMutableMetrics
            org.apache.hadoop.security.TestLdapGroupsMapping
            org.apache.hadoop.security.token.delegation.web.TestDelegationTokenManager
            org.apache.hadoop.ipc.TestSaslRPC
            org.apache.hadoop.net.TestScriptBasedMapping
            org.apache.hadoop.net.TestSwitchMapping
            org.apache.hadoop.fs.TestFcLocalFsPermission
            org.apache.hadoop.fs.viewfs.TestChRootedFs
            org.apache.hadoop.fs.viewfs.TestViewFsLocalFs
            org.apache.hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem
            org.apache.hadoop.fs.TestFileContextDeleteOnExit
            org.apache.hadoop.fs.shell.find.TestIname
            org.apache.hadoop.fs.shell.find.TestPrint0
            org.apache.hadoop.fs.shell.find.TestFind
            org.apache.hadoop.fs.shell.TestCopy
            org.apache.hadoop.fs.shell.TestCopyPreserveFlag
            org.apache.hadoop.fs.shell.TestTextCommand
            org.apache.hadoop.fs.TestFsShell
            org.apache.hadoop.fs.TestLocalFSFileContextCreateMkdir
            org.apache.hadoop.fs.TestSymlinkLocalFSFileContext
            org.apache.hadoop.fs.TestFSMainOperationsLocalFileSystem
            org.apache.hadoop.fs.TestS3_LocalFileContextURI
            org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractAppend
            org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractCreate
            org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractOpen
            org.apache.hadoop.fs.contract.localfs.TestLocalFSContractRename
            org.apache.hadoop.fs.contract.localfs.TestLocalFSContractAppend
            org.apache.hadoop.fs.contract.localfs.TestLocalFSContractCreate
            org.apache.hadoop.fs.contract.ftp.TestFTPContractOpen
            org.apache.hadoop.fs.contract.ftp.TestFTPContractCreate
            org.apache.hadoop.fs.TestLocal_S3FileContextURI
            org.apache.hadoop.fs.TestHarFileSystemBasics
            org.apache.hadoop.fs.TestDelegationTokenRenewer
            org.apache.hadoop.util.TestApplicationClassLoader
            org.apache.hadoop.util.TestDataChecksum
            org.apache.hadoop.util.bloom.TestBloomFilters
            org.apache.hadoop.util.TestAsyncDiskService
            org.apache.hadoop.util.TestStringUtils
            org.apache.hadoop.util.TestLightWeightGSet
            org.apache.hadoop.util.TestFindClass
            org.apache.hadoop.util.TestClassUtil
            org.apache.hadoop.util.TestLightWeightCache
            org.apache.hadoop.util.TestWinUtils
            org.apache.hadoop.util.TestJarFinder
            org.apache.hadoop.util.TestClasspath
            org.apache.hadoop.http.TestHttpServer
            org.apache.hadoop.http.TestHttpServerWebapps
            org.apache.hadoop.crypto.TestCryptoCodec
            org.apache.hadoop.crypto.key.TestKeyProvider
            org.apache.hadoop.io.serializer.TestSerializationFactory
            org.apache.hadoop.io.serializer.TestWritableSerialization
            org.apache.hadoop.io.compress.TestCodecPool
            org.apache.hadoop.io.compress.TestCompressorDecompressor
            org.apache.hadoop.io.TestSequenceFileSync
            org.apache.hadoop.io.TestBloomMapFile
            org.apache.hadoop.io.TestBytesWritable
            org.apache.hadoop.io.TestTextNonUTF8
            org.apache.hadoop.io.retry.TestRetryProxy
            org.apache.hadoop.io.TestIOUtils
            org.apache.hadoop.io.TestArrayPrimitiveWritable
            org.apache.hadoop.io.file.tfile.TestTFileStreams
            org.apache.hadoop.io.file.tfile.TestTFileByteArrays
            org.apache.hadoop.conf.TestReconfiguration



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12733326/HADOOP-11984.002.patch
          Optional Tests shellcheck javac unit findbugs checkstyle
          git revision trunk / b0ad644
          shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/6714/artifact/patchprocess/diffpatchshellcheck.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6714/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6714/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6714/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 5m 11s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 31s There were no new javac warning messages. +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 3s There were no new checkstyle issues. -1 shellcheck 0m 8s The applied patch generated 13 new shellcheck (v0.3.3) issues (total was 38, now 51). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 37s The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 common tests 20m 33s Tests failed in hadoop-common.     38m 29s   Reason Tests Failed unit tests hadoop.fs.contract.localfs.TestLocalFSContractOpen   hadoop.fs.contract.localfs.TestLocalFSContractRename   hadoop.fs.contract.localfs.TestLocalFSContractMkdir   hadoop.ha.TestZKFailoverControllerStress   hadoop.fs.contract.localfs.TestLocalFSContractDelete   hadoop.fs.contract.localfs.TestLocalFSContractCreate Timed out tests org.apache.hadoop.service.TestServiceLifecycle   org.apache.hadoop.service.TestCompositeService   org.apache.hadoop.service.TestGlobalStateChangeListener   org.apache.hadoop.ha.TestActiveStandbyElectorRealZK   org.apache.hadoop.metrics.TestMetricsServlet   org.apache.hadoop.metrics2.impl.TestMetricsVisitor   org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl   org.apache.hadoop.metrics2.impl.TestGangliaMetrics   org.apache.hadoop.metrics2.lib.TestMetricsRegistry   org.apache.hadoop.metrics2.lib.TestMutableMetrics   org.apache.hadoop.security.TestLdapGroupsMapping   org.apache.hadoop.security.token.delegation.web.TestDelegationTokenManager   org.apache.hadoop.ipc.TestSaslRPC   org.apache.hadoop.net.TestScriptBasedMapping   org.apache.hadoop.net.TestSwitchMapping   org.apache.hadoop.fs.TestFcLocalFsPermission   org.apache.hadoop.fs.viewfs.TestChRootedFs   org.apache.hadoop.fs.viewfs.TestViewFsLocalFs   org.apache.hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem   org.apache.hadoop.fs.TestFileContextDeleteOnExit   org.apache.hadoop.fs.shell.find.TestIname   org.apache.hadoop.fs.shell.find.TestPrint0   org.apache.hadoop.fs.shell.find.TestFind   org.apache.hadoop.fs.shell.TestCopy   org.apache.hadoop.fs.shell.TestCopyPreserveFlag   org.apache.hadoop.fs.shell.TestTextCommand   org.apache.hadoop.fs.TestFsShell   org.apache.hadoop.fs.TestLocalFSFileContextCreateMkdir   org.apache.hadoop.fs.TestSymlinkLocalFSFileContext   org.apache.hadoop.fs.TestFSMainOperationsLocalFileSystem   org.apache.hadoop.fs.TestS3_LocalFileContextURI   org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractAppend   org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractCreate   org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractOpen   org.apache.hadoop.fs.contract.localfs.TestLocalFSContractRename   org.apache.hadoop.fs.contract.localfs.TestLocalFSContractAppend   org.apache.hadoop.fs.contract.localfs.TestLocalFSContractCreate   org.apache.hadoop.fs.contract.ftp.TestFTPContractOpen   org.apache.hadoop.fs.contract.ftp.TestFTPContractCreate   org.apache.hadoop.fs.TestLocal_S3FileContextURI   org.apache.hadoop.fs.TestHarFileSystemBasics   org.apache.hadoop.fs.TestDelegationTokenRenewer   org.apache.hadoop.util.TestApplicationClassLoader   org.apache.hadoop.util.TestDataChecksum   org.apache.hadoop.util.bloom.TestBloomFilters   org.apache.hadoop.util.TestAsyncDiskService   org.apache.hadoop.util.TestStringUtils   org.apache.hadoop.util.TestLightWeightGSet   org.apache.hadoop.util.TestFindClass   org.apache.hadoop.util.TestClassUtil   org.apache.hadoop.util.TestLightWeightCache   org.apache.hadoop.util.TestWinUtils   org.apache.hadoop.util.TestJarFinder   org.apache.hadoop.util.TestClasspath   org.apache.hadoop.http.TestHttpServer   org.apache.hadoop.http.TestHttpServerWebapps   org.apache.hadoop.crypto.TestCryptoCodec   org.apache.hadoop.crypto.key.TestKeyProvider   org.apache.hadoop.io.serializer.TestSerializationFactory   org.apache.hadoop.io.serializer.TestWritableSerialization   org.apache.hadoop.io.compress.TestCodecPool   org.apache.hadoop.io.compress.TestCompressorDecompressor   org.apache.hadoop.io.TestSequenceFileSync   org.apache.hadoop.io.TestBloomMapFile   org.apache.hadoop.io.TestBytesWritable   org.apache.hadoop.io.TestTextNonUTF8   org.apache.hadoop.io.retry.TestRetryProxy   org.apache.hadoop.io.TestIOUtils   org.apache.hadoop.io.TestArrayPrimitiveWritable   org.apache.hadoop.io.file.tfile.TestTFileStreams   org.apache.hadoop.io.file.tfile.TestTFileByteArrays   org.apache.hadoop.conf.TestReconfiguration Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733326/HADOOP-11984.002.patch Optional Tests shellcheck javac unit findbugs checkstyle git revision trunk / b0ad644 shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/6714/artifact/patchprocess/diffpatchshellcheck.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6714/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6714/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6714/console This message was automatically generated.
          Hide
          wheat9 Haohui Mai added a comment -

          Please correct me if I'm wrong – Looks like that test-patch.sh relies on a lot of regex to do the job. Does it make sense to move some of these work towards a Python script so that it is possible to leverage more libraries? For example, To determine whether a test run successfully, it seems more reliable to parse the xml output of test result rather than doing a regex matching.

          Show
          wheat9 Haohui Mai added a comment - Please correct me if I'm wrong – Looks like that test-patch.sh relies on a lot of regex to do the job. Does it make sense to move some of these work towards a Python script so that it is possible to leverage more libraries? For example, To determine whether a test run successfully, it seems more reliable to parse the xml output of test result rather than doing a regex matching.
          Hide
          cnauroth Chris Nauroth added a comment -

          We saw OutOfMemoryError in that hadoop-common run too. We'll need to take a closer look at how Surefire allocates heap size for multiple concurrent JUnit processes.

          A few of the file system contract tests failed, and I think it's because they collided on the test.build.data directory. In hadoop-hdfs, we made a pom.xml change to parameterize that directory based on the JUnit process number. That isn't done in the hadoop-common pom.xml, so we'll need to change it.

                        <systemPropertyVariables>
                          <test.build.data>${test.build.data}/${surefire.forkNumber}</test.build.data>
                          <hadoop.tmp.dir>${hadoop.tmp.dir}/${surefire.forkNumber}</hadoop.tmp.dir>
                        </systemPropertyVariables>
          

          I'm not sure about TestZKFailoverControllerStress yet, although that one has been flaky in the past.

          Does it make sense to move some of these work towards a Python script so that it is possible to leverage more libraries?

          I didn't have any particular solution in mind, so I think anything is on the table at this point. If we can find another heuristic for identifying timed out tests using bash/grep/awk, then I think that's fine. If we can't find a reliable heuristic, then parsing the XML sounds good.

          Show
          cnauroth Chris Nauroth added a comment - We saw OutOfMemoryError in that hadoop-common run too. We'll need to take a closer look at how Surefire allocates heap size for multiple concurrent JUnit processes. A few of the file system contract tests failed, and I think it's because they collided on the test.build.data directory. In hadoop-hdfs, we made a pom.xml change to parameterize that directory based on the JUnit process number. That isn't done in the hadoop-common pom.xml, so we'll need to change it. <systemPropertyVariables> <test.build.data>${test.build.data}/${surefire.forkNumber}</test.build.data> <hadoop.tmp.dir>${hadoop.tmp.dir}/${surefire.forkNumber}</hadoop.tmp.dir> </systemPropertyVariables> I'm not sure about TestZKFailoverControllerStress yet, although that one has been flaky in the past. Does it make sense to move some of these work towards a Python script so that it is possible to leverage more libraries? I didn't have any particular solution in mind, so I think anything is on the table at this point. If we can find another heuristic for identifying timed out tests using bash/grep/awk, then I think that's fine. If we can't find a reliable heuristic, then parsing the XML sounds good.
          Hide
          cnauroth Chris Nauroth added a comment -

          Patch v003 refactors the definition of the parallel-tests profile into hadoop-project. That fixes the prior failures in the file system contract tests, because we now parameterize the test data directory by JUnit process number.

          Show
          cnauroth Chris Nauroth added a comment - Patch v003 refactors the definition of the parallel-tests profile into hadoop-project. That fixes the prior failures in the file system contract tests, because we now parameterize the test data directory by JUnit process number.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to test-patch or smart-apply-patch has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6715/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6715/console in case of problems.
          Hide
          cnauroth Chris Nauroth added a comment -

          Patch v004 sets the reuseForks Surefire property to false. TestConfiguration runs a background thread that keeps allocating memory, which gradually can cause OutOfMemoryError for subsequent test suites. Static initialization also could be a problem if the forked JVM processes were reused.

          Show
          cnauroth Chris Nauroth added a comment - Patch v004 sets the reuseForks Surefire property to false. TestConfiguration runs a background thread that keeps allocating memory, which gradually can cause OutOfMemoryError for subsequent test suites. Static initialization also could be a problem if the forked JVM processes were reused.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to test-patch or smart-apply-patch has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6716/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6716/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 14m 31s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 30s There were no new javac warning messages.
          +1 javadoc 9m 33s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 3m 27s There were no new checkstyle issues.
          +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
          +1 findbugs 4m 42s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          -1 common tests 19m 36s Tests failed in hadoop-common.
          -1 hdfs tests 45m 51s Tests failed in hadoop-hdfs.
              107m 49s  



          Reason Tests
          Failed unit tests hadoop.ha.TestZKFailoverController
            hadoop.security.TestCredentials
            hadoop.ha.TestActiveStandbyElectorRealZK
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
          Timed out tests org.apache.hadoop.log.TestLogLevel
            org.apache.hadoop.service.TestServiceLifecycle
            org.apache.hadoop.log.TestLog4Json
            org.apache.hadoop.service.TestCompositeService
            org.apache.hadoop.security.TestDoAsEffectiveUser
            org.apache.hadoop.security.TestUserFromEnv
            org.apache.hadoop.security.TestGroupFallback
            org.apache.hadoop.security.TestJNIGroupsMapping
            org.apache.hadoop.security.ssl.TestSSLFactory
            org.apache.hadoop.security.TestAuthenticationFilter
            org.apache.hadoop.security.token.delegation.web.TestDelegationTokenManager
            org.apache.hadoop.security.token.delegation.web.TestDelegationTokenAuthenticationHandlerWithMocks
            org.apache.hadoop.ha.TestFailoverController
            org.apache.hadoop.http.TestGlobalFilter
            org.apache.hadoop.http.TestHtmlQuoting
            org.apache.hadoop.util.TestDiskChecker
            org.apache.hadoop.util.hash.TestHash
            org.apache.hadoop.util.curator.TestChildReaper
            org.apache.hadoop.util.TestNativeCrc32
            org.apache.hadoop.util.TestStringInterner
            org.apache.hadoop.util.TestIndexedSort
            org.apache.hadoop.util.TestSignalLogger
            org.apache.hadoop.util.TestApplicationClassLoader
            org.apache.hadoop.util.TestReflectionUtils
            org.apache.hadoop.ipc.TestSaslRPC
            org.apache.hadoop.fs.TestGetFileBlockLocations
            org.apache.hadoop.fs.contract.localfs.TestLocalFSContractAppend
            org.apache.hadoop.fs.contract.localfs.TestLocalFSContractSeek
            org.apache.hadoop.fs.contract.localfs.TestLocalFSContractRename
            org.apache.hadoop.fs.contract.ftp.TestFTPContractCreate
            org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractAppend
            org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractCreate
            org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractRename
            org.apache.hadoop.fs.permission.TestAcl
            org.apache.hadoop.fs.permission.TestFsPermission
            org.apache.hadoop.fs.TestAfsCheckPath
            org.apache.hadoop.fs.shell.find.TestPrint0
            org.apache.hadoop.fs.shell.find.TestResult
            org.apache.hadoop.fs.shell.TestCopyPreserveFlag
            org.apache.hadoop.fs.TestFcLocalFsPermission
            org.apache.hadoop.fs.TestFileUtil
            org.apache.hadoop.fs.TestFSMainOperationsLocalFileSystem
            org.apache.hadoop.fs.TestFileContextDeleteOnExit
            org.apache.hadoop.fs.viewfs.TestFcPermissionsLocalFs
            org.apache.hadoop.fs.viewfs.TestViewFsTrash
            org.apache.hadoop.fs.viewfs.TestViewFsWithAuthorityLocalFs
            org.apache.hadoop.fs.TestLocalFSFileContextCreateMkdir
            org.apache.hadoop.fs.TestHardLink
            org.apache.hadoop.fs.TestLocalFileSystem
            org.apache.hadoop.fs.TestFsShell
            org.apache.hadoop.crypto.random.TestOsSecureRandom
            org.apache.hadoop.crypto.key.TestKeyProviderDelegationTokenExtension
            org.apache.hadoop.crypto.key.TestValueQueue
            org.apache.hadoop.crypto.TestCryptoStreamsNormal
            org.apache.hadoop.crypto.TestCryptoCodec
            org.apache.hadoop.metrics2.util.TestMetricsCache
            org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
            org.apache.hadoop.metrics2.impl.TestMetricsSourceAdapter
            org.apache.hadoop.metrics2.lib.TestMutableMetrics
            org.apache.hadoop.metrics2.source.TestJvmMetrics
            org.apache.hadoop.conf.TestConfigurationDeprecation
            org.apache.hadoop.conf.TestDeprecatedKeys
            org.apache.hadoop.net.TestNetUtils
            org.apache.hadoop.net.TestScriptBasedMapping
            org.apache.hadoop.net.unix.TestDomainSocket
            org.apache.hadoop.io.TestIOUtils
            org.apache.hadoop.io.TestEnumSetWritable
            org.apache.hadoop.io.TestBytesWritable
            org.apache.hadoop.io.file.tfile.TestTFileComparators
            org.apache.hadoop.io.file.tfile.TestTFileNoneCodecsJClassComparatorByteArrays
            org.apache.hadoop.io.file.tfile.TestTFileNoneCodecsStreams
            org.apache.hadoop.io.file.tfile.TestTFileByteArrays
            org.apache.hadoop.io.TestArrayWritable
            org.apache.hadoop.io.TestText
            org.apache.hadoop.io.compress.TestCompressionStreamReuse
            org.apache.hadoop.io.TestSecureIOUtils
            org.apache.hadoop.io.nativeio.TestNativeIO
            org.apache.hadoop.hdfs.TestDFSInputStream
            org.apache.hadoop.hdfs.TestDatanodeRegistration
            org.apache.hadoop.hdfs.TestPipelines
            org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache
            org.apache.hadoop.hdfs.server.datanode.TestCachingStrategy
            org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner
            org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCacheRevocation
            org.apache.hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport
            org.apache.hadoop.hdfs.server.datanode.TestDataNodeECN
            org.apache.hadoop.hdfs.server.datanode.TestDataNodeInitStorage
            org.apache.hadoop.hdfs.server.namenode.TestHDFSConcat
            org.apache.hadoop.hdfs.server.namenode.TestGetImageServlet
            org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete
            org.apache.hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot
            org.apache.hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots
            org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives
            org.apache.hadoop.hdfs.server.namenode.ha.TestHAConfiguration
            org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
            org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogsDuringFailover
            org.apache.hadoop.hdfs.server.namenode.TestMetadataVersionOutput
            org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks
            org.apache.hadoop.hdfs.server.namenode.TestCheckpoint
            org.apache.hadoop.hdfs.server.namenode.TestAuditLogger
            org.apache.hadoop.hdfs.server.namenode.TestNamenodeRetryCache
            org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery
            org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure
            org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs
            org.apache.hadoop.hdfs.web.TestWebHdfsContentLength
            org.apache.hadoop.hdfs.web.TestByteRangeInputStream
            org.apache.hadoop.hdfs.TestBlockStoragePolicy
            org.apache.hadoop.hdfs.crypto.TestHdfsCryptoStreams
            org.apache.hadoop.hdfs.TestMiniDFSCluster
            org.apache.hadoop.hdfs.tools.TestDebugAdmin
            org.apache.hadoop.hdfs.qjournal.server.TestJournal
            org.apache.hadoop.hdfs.TestReadWhileWriting
            org.apache.hadoop.hdfs.TestCrcCorruption
            org.apache.hadoop.hdfs.TestLocalDFS
            org.apache.hadoop.hdfs.TestDecommission
            org.apache.hadoop.fs.contract.hdfs.TestHDFSContractDelete
            org.apache.hadoop.fs.contract.hdfs.TestHDFSContractConcat
            org.apache.hadoop.fs.contract.hdfs.TestHDFSContractRootDirectory
            org.apache.hadoop.fs.TestEnhancedByteBufferAccess
            org.apache.hadoop.fs.permission.TestStickyBit
            org.apache.hadoop.fs.TestResolveHdfsSymlink
            org.apache.hadoop.fs.TestFcHdfsCreateMkdir
            org.apache.hadoop.fs.TestUrlStreamHandler
            org.apache.hadoop.fs.viewfs.TestViewFsWithAcls
            org.apache.hadoop.fs.viewfs.TestViewFileSystemWithAcls
            org.apache.hadoop.fs.viewfs.TestViewFsWithXAttrs
            org.apache.hadoop.cli.TestCacheAdminCLI



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12733350/HADOOP-11984.003.patch
          Optional Tests shellcheck javadoc javac unit findbugs checkstyle
          git revision trunk / e453989
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6715/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6715/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6715/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6715/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 14m 31s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 33s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 3m 27s There were no new checkstyle issues. +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 findbugs 4m 42s The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 common tests 19m 36s Tests failed in hadoop-common. -1 hdfs tests 45m 51s Tests failed in hadoop-hdfs.     107m 49s   Reason Tests Failed unit tests hadoop.ha.TestZKFailoverController   hadoop.security.TestCredentials   hadoop.ha.TestActiveStandbyElectorRealZK   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery Timed out tests org.apache.hadoop.log.TestLogLevel   org.apache.hadoop.service.TestServiceLifecycle   org.apache.hadoop.log.TestLog4Json   org.apache.hadoop.service.TestCompositeService   org.apache.hadoop.security.TestDoAsEffectiveUser   org.apache.hadoop.security.TestUserFromEnv   org.apache.hadoop.security.TestGroupFallback   org.apache.hadoop.security.TestJNIGroupsMapping   org.apache.hadoop.security.ssl.TestSSLFactory   org.apache.hadoop.security.TestAuthenticationFilter   org.apache.hadoop.security.token.delegation.web.TestDelegationTokenManager   org.apache.hadoop.security.token.delegation.web.TestDelegationTokenAuthenticationHandlerWithMocks   org.apache.hadoop.ha.TestFailoverController   org.apache.hadoop.http.TestGlobalFilter   org.apache.hadoop.http.TestHtmlQuoting   org.apache.hadoop.util.TestDiskChecker   org.apache.hadoop.util.hash.TestHash   org.apache.hadoop.util.curator.TestChildReaper   org.apache.hadoop.util.TestNativeCrc32   org.apache.hadoop.util.TestStringInterner   org.apache.hadoop.util.TestIndexedSort   org.apache.hadoop.util.TestSignalLogger   org.apache.hadoop.util.TestApplicationClassLoader   org.apache.hadoop.util.TestReflectionUtils   org.apache.hadoop.ipc.TestSaslRPC   org.apache.hadoop.fs.TestGetFileBlockLocations   org.apache.hadoop.fs.contract.localfs.TestLocalFSContractAppend   org.apache.hadoop.fs.contract.localfs.TestLocalFSContractSeek   org.apache.hadoop.fs.contract.localfs.TestLocalFSContractRename   org.apache.hadoop.fs.contract.ftp.TestFTPContractCreate   org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractAppend   org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractCreate   org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractRename   org.apache.hadoop.fs.permission.TestAcl   org.apache.hadoop.fs.permission.TestFsPermission   org.apache.hadoop.fs.TestAfsCheckPath   org.apache.hadoop.fs.shell.find.TestPrint0   org.apache.hadoop.fs.shell.find.TestResult   org.apache.hadoop.fs.shell.TestCopyPreserveFlag   org.apache.hadoop.fs.TestFcLocalFsPermission   org.apache.hadoop.fs.TestFileUtil   org.apache.hadoop.fs.TestFSMainOperationsLocalFileSystem   org.apache.hadoop.fs.TestFileContextDeleteOnExit   org.apache.hadoop.fs.viewfs.TestFcPermissionsLocalFs   org.apache.hadoop.fs.viewfs.TestViewFsTrash   org.apache.hadoop.fs.viewfs.TestViewFsWithAuthorityLocalFs   org.apache.hadoop.fs.TestLocalFSFileContextCreateMkdir   org.apache.hadoop.fs.TestHardLink   org.apache.hadoop.fs.TestLocalFileSystem   org.apache.hadoop.fs.TestFsShell   org.apache.hadoop.crypto.random.TestOsSecureRandom   org.apache.hadoop.crypto.key.TestKeyProviderDelegationTokenExtension   org.apache.hadoop.crypto.key.TestValueQueue   org.apache.hadoop.crypto.TestCryptoStreamsNormal   org.apache.hadoop.crypto.TestCryptoCodec   org.apache.hadoop.metrics2.util.TestMetricsCache   org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl   org.apache.hadoop.metrics2.impl.TestMetricsSourceAdapter   org.apache.hadoop.metrics2.lib.TestMutableMetrics   org.apache.hadoop.metrics2.source.TestJvmMetrics   org.apache.hadoop.conf.TestConfigurationDeprecation   org.apache.hadoop.conf.TestDeprecatedKeys   org.apache.hadoop.net.TestNetUtils   org.apache.hadoop.net.TestScriptBasedMapping   org.apache.hadoop.net.unix.TestDomainSocket   org.apache.hadoop.io.TestIOUtils   org.apache.hadoop.io.TestEnumSetWritable   org.apache.hadoop.io.TestBytesWritable   org.apache.hadoop.io.file.tfile.TestTFileComparators   org.apache.hadoop.io.file.tfile.TestTFileNoneCodecsJClassComparatorByteArrays   org.apache.hadoop.io.file.tfile.TestTFileNoneCodecsStreams   org.apache.hadoop.io.file.tfile.TestTFileByteArrays   org.apache.hadoop.io.TestArrayWritable   org.apache.hadoop.io.TestText   org.apache.hadoop.io.compress.TestCompressionStreamReuse   org.apache.hadoop.io.TestSecureIOUtils   org.apache.hadoop.io.nativeio.TestNativeIO   org.apache.hadoop.hdfs.TestDFSInputStream   org.apache.hadoop.hdfs.TestDatanodeRegistration   org.apache.hadoop.hdfs.TestPipelines   org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache   org.apache.hadoop.hdfs.server.datanode.TestCachingStrategy   org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner   org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCacheRevocation   org.apache.hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport   org.apache.hadoop.hdfs.server.datanode.TestDataNodeECN   org.apache.hadoop.hdfs.server.datanode.TestDataNodeInitStorage   org.apache.hadoop.hdfs.server.namenode.TestHDFSConcat   org.apache.hadoop.hdfs.server.namenode.TestGetImageServlet   org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete   org.apache.hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot   org.apache.hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots   org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives   org.apache.hadoop.hdfs.server.namenode.ha.TestHAConfiguration   org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogsDuringFailover   org.apache.hadoop.hdfs.server.namenode.TestMetadataVersionOutput   org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks   org.apache.hadoop.hdfs.server.namenode.TestCheckpoint   org.apache.hadoop.hdfs.server.namenode.TestAuditLogger   org.apache.hadoop.hdfs.server.namenode.TestNamenodeRetryCache   org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery   org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure   org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs   org.apache.hadoop.hdfs.web.TestWebHdfsContentLength   org.apache.hadoop.hdfs.web.TestByteRangeInputStream   org.apache.hadoop.hdfs.TestBlockStoragePolicy   org.apache.hadoop.hdfs.crypto.TestHdfsCryptoStreams   org.apache.hadoop.hdfs.TestMiniDFSCluster   org.apache.hadoop.hdfs.tools.TestDebugAdmin   org.apache.hadoop.hdfs.qjournal.server.TestJournal   org.apache.hadoop.hdfs.TestReadWhileWriting   org.apache.hadoop.hdfs.TestCrcCorruption   org.apache.hadoop.hdfs.TestLocalDFS   org.apache.hadoop.hdfs.TestDecommission   org.apache.hadoop.fs.contract.hdfs.TestHDFSContractDelete   org.apache.hadoop.fs.contract.hdfs.TestHDFSContractConcat   org.apache.hadoop.fs.contract.hdfs.TestHDFSContractRootDirectory   org.apache.hadoop.fs.TestEnhancedByteBufferAccess   org.apache.hadoop.fs.permission.TestStickyBit   org.apache.hadoop.fs.TestResolveHdfsSymlink   org.apache.hadoop.fs.TestFcHdfsCreateMkdir   org.apache.hadoop.fs.TestUrlStreamHandler   org.apache.hadoop.fs.viewfs.TestViewFsWithAcls   org.apache.hadoop.fs.viewfs.TestViewFileSystemWithAcls   org.apache.hadoop.fs.viewfs.TestViewFsWithXAttrs   org.apache.hadoop.cli.TestCacheAdminCLI Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733350/HADOOP-11984.003.patch Optional Tests shellcheck javadoc javac unit findbugs checkstyle git revision trunk / e453989 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6715/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6715/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6715/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6715/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 14m 32s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 32s There were no new javac warning messages.
          +1 javadoc 9m 36s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          -1 common tests 18m 1s Tests failed in hadoop-common.
          -1 hdfs tests 41m 41s Tests failed in hadoop-hdfs.
              94m 2s  



          Reason Tests
          Failed unit tests hadoop.ha.TestActiveStandbyElectorRealZK
            hadoop.ha.TestZKFailoverControllerStress
            hadoop.ha.TestZKFailoverController
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.TestAppendSnapshotTruncate
          Timed out tests org.apache.hadoop.service.TestServiceLifecycle
            org.apache.hadoop.service.TestCompositeService
            org.apache.hadoop.service.TestGlobalStateChangeListener
            org.apache.hadoop.ha.TestNodeFencer
            org.apache.hadoop.metrics.ganglia.TestGangliaContext
            org.apache.hadoop.metrics.spi.TestOutputRecord
            org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
            org.apache.hadoop.metrics2.impl.TestGangliaMetrics
            org.apache.hadoop.metrics2.lib.TestMetricsRegistry
            org.apache.hadoop.metrics2.lib.TestMutableMetrics
            org.apache.hadoop.security.authorize.TestProxyServers
            org.apache.hadoop.security.authorize.TestProxyUsers
            org.apache.hadoop.ipc.TestRetryCacheMetrics
            org.apache.hadoop.ipc.TestServer
            org.apache.hadoop.net.unix.TestDomainSocket
            org.apache.hadoop.net.TestSwitchMapping
            org.apache.hadoop.fs.TestAvroFSInput
            org.apache.hadoop.fs.TestFcLocalFsPermission
            org.apache.hadoop.fs.viewfs.TestChRootedFs
            org.apache.hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem
            org.apache.hadoop.fs.TestContentSummary
            org.apache.hadoop.fs.TestFileContextDeleteOnExit
            org.apache.hadoop.fs.shell.find.TestIname
            org.apache.hadoop.fs.shell.find.TestPrint0
            org.apache.hadoop.fs.shell.find.TestFind
            org.apache.hadoop.fs.shell.TestCopy
            org.apache.hadoop.fs.shell.TestCount
            org.apache.hadoop.fs.shell.TestCopyPreserveFlag
            org.apache.hadoop.fs.shell.TestXAttrCommands
            org.apache.hadoop.fs.TestFsShell
            org.apache.hadoop.fs.TestLocalFSFileContextCreateMkdir
            org.apache.hadoop.fs.TestSymlinkLocalFSFileContext
            org.apache.hadoop.fs.TestS3_LocalFileContextURI
            org.apache.hadoop.fs.TestFileSystemTokens
            org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractAppend
            org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractCreate
            org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractOpen
            org.apache.hadoop.fs.contract.localfs.TestLocalFSContractRename
            org.apache.hadoop.fs.contract.localfs.TestLocalFSContractAppend
            org.apache.hadoop.fs.contract.localfs.TestLocalFSContractCreate
            org.apache.hadoop.fs.contract.ftp.TestFTPContractRename
            org.apache.hadoop.fs.contract.ftp.TestFTPContractOpen
            org.apache.hadoop.fs.contract.ftp.TestFTPContractMkdir
            org.apache.hadoop.fs.contract.ftp.TestFTPContractCreate
            org.apache.hadoop.fs.TestLocalFSFileContextMainOperations
            org.apache.hadoop.fs.permission.TestFsPermission
            org.apache.hadoop.fs.TestHarFileSystemBasics
            org.apache.hadoop.fs.TestStat
            org.apache.hadoop.fs.TestPath
            org.apache.hadoop.util.TestHttpExceptionUtils
            org.apache.hadoop.util.TestHostsFileReader
            org.apache.hadoop.util.TestChunkedArrayList
            org.apache.hadoop.util.TestLightWeightGSet
            org.apache.hadoop.util.TestGenericOptionsParser
            org.apache.hadoop.util.TestNativeCodeLoader
            org.apache.hadoop.util.TestNativeLibraryChecker
            org.apache.hadoop.util.TestLightWeightCache
            org.apache.hadoop.util.TestWinUtils
            org.apache.hadoop.util.TestProtoUtil
            org.apache.hadoop.util.TestDirectBufferPool
            org.apache.hadoop.util.TestGSet
            org.apache.hadoop.http.TestPathFilter
            org.apache.hadoop.http.TestHttpServer
            org.apache.hadoop.http.TestHttpServerWebapps
            org.apache.hadoop.crypto.TestCryptoCodec
            org.apache.hadoop.crypto.key.TestKeyShell
            org.apache.hadoop.io.TestWritableUtils
            org.apache.hadoop.io.TestSequenceFileSync
            org.apache.hadoop.io.TestWritable
            org.apache.hadoop.io.TestEnumSetWritable
            org.apache.hadoop.io.TestSequenceFile
            org.apache.hadoop.io.TestTextNonUTF8
            org.apache.hadoop.io.TestDefaultStringifier
            org.apache.hadoop.io.TestArrayPrimitiveWritable
            org.apache.hadoop.io.file.tfile.TestTFileComparators
            org.apache.hadoop.io.file.tfile.TestTFileNoneCodecsByteArrays
            org.apache.hadoop.io.file.tfile.TestTFileByteArrays
            org.apache.hadoop.io.file.tfile.TestTFileLzoCodecsStreams
            org.apache.hadoop.io.file.tfile.TestTFileComparator2
            org.apache.hadoop.conf.TestReconfiguration
            org.apache.hadoop.log.TestLog4Json
            org.apache.hadoop.TestGenericRefresh
            org.apache.hadoop.TestRefreshCallQueue
            org.apache.hadoop.cli.TestCryptoAdminCLI
            org.apache.hadoop.cli.TestHDFSCLI
            org.apache.hadoop.cli.TestXAttrCLI
            org.apache.hadoop.fs.viewfs.TestViewFileSystemWithXAttrs
            org.apache.hadoop.fs.viewfs.TestViewFileSystemWithAcls
            org.apache.hadoop.fs.shell.TestHdfsTextCommand
            org.apache.hadoop.fs.contract.hdfs.TestHDFSContractSeek
            org.apache.hadoop.fs.loadGenerator.TestLoadGenerator
            org.apache.hadoop.fs.TestSymlinkHdfsFileSystem
            org.apache.hadoop.tools.TestJMXGet
            org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached
            org.apache.hadoop.hdfs.qjournal.server.TestJournalNodeMXBean
            org.apache.hadoop.hdfs.qjournal.client.TestSegmentRecoveryComparator
            org.apache.hadoop.hdfs.qjournal.client.TestQuorumCall
            org.apache.hadoop.hdfs.TestWriteRead
            org.apache.hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN
            org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool
            org.apache.hadoop.hdfs.server.datanode.TestStartSecureDataNode
            org.apache.hadoop.hdfs.server.namenode.TestFileLimit
            org.apache.hadoop.hdfs.server.namenode.TestFSImageWithSnapshot
            org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
            org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogsDuringFailover
            org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
            org.apache.hadoop.hdfs.server.namenode.TestCheckpoint
            org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics
            org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters
            org.apache.hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics
            org.apache.hadoop.hdfs.server.namenode.TestEditLogFileInputStream
            org.apache.hadoop.hdfs.server.namenode.TestSecondaryWebUi
            org.apache.hadoop.hdfs.server.balancer.TestBalancer
            org.apache.hadoop.hdfs.web.TestWebHdfsWithAuthenticationFilter
            org.apache.hadoop.hdfs.TestFsShellPermission
            org.apache.hadoop.hdfs.TestRollingUpgrade
            org.apache.hadoop.hdfs.TestParallelShortCircuitReadNoChecksum
            org.apache.hadoop.hdfs.TestDFSRollback
            org.apache.hadoop.hdfs.tools.TestDFSHAAdmin
            org.apache.hadoop.hdfs.tools.TestDFSHAAdminMiniCluster
            org.apache.hadoop.hdfs.TestBlockStoragePolicy
            org.apache.hadoop.hdfs.TestCrcCorruption
            org.apache.hadoop.hdfs.TestWriteConfigurationToDFS
            org.apache.hadoop.hdfs.TestDataTransferKeepalive
            org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
            org.apache.hadoop.hdfs.TestSnapshotCommands
            org.apache.hadoop.hdfs.TestRollingUpgradeDowngrade
            org.apache.hadoop.hdfs.TestGetBlocks
            org.apache.hadoop.hdfs.TestReplication



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12733351/HADOOP-11984.004.patch
          Optional Tests shellcheck javadoc javac unit
          git revision trunk / e453989
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6716/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6716/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6716/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6716/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 14m 32s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 32s There were no new javac warning messages. +1 javadoc 9m 36s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 common tests 18m 1s Tests failed in hadoop-common. -1 hdfs tests 41m 41s Tests failed in hadoop-hdfs.     94m 2s   Reason Tests Failed unit tests hadoop.ha.TestActiveStandbyElectorRealZK   hadoop.ha.TestZKFailoverControllerStress   hadoop.ha.TestZKFailoverController   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.TestAppendSnapshotTruncate Timed out tests org.apache.hadoop.service.TestServiceLifecycle   org.apache.hadoop.service.TestCompositeService   org.apache.hadoop.service.TestGlobalStateChangeListener   org.apache.hadoop.ha.TestNodeFencer   org.apache.hadoop.metrics.ganglia.TestGangliaContext   org.apache.hadoop.metrics.spi.TestOutputRecord   org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl   org.apache.hadoop.metrics2.impl.TestGangliaMetrics   org.apache.hadoop.metrics2.lib.TestMetricsRegistry   org.apache.hadoop.metrics2.lib.TestMutableMetrics   org.apache.hadoop.security.authorize.TestProxyServers   org.apache.hadoop.security.authorize.TestProxyUsers   org.apache.hadoop.ipc.TestRetryCacheMetrics   org.apache.hadoop.ipc.TestServer   org.apache.hadoop.net.unix.TestDomainSocket   org.apache.hadoop.net.TestSwitchMapping   org.apache.hadoop.fs.TestAvroFSInput   org.apache.hadoop.fs.TestFcLocalFsPermission   org.apache.hadoop.fs.viewfs.TestChRootedFs   org.apache.hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem   org.apache.hadoop.fs.TestContentSummary   org.apache.hadoop.fs.TestFileContextDeleteOnExit   org.apache.hadoop.fs.shell.find.TestIname   org.apache.hadoop.fs.shell.find.TestPrint0   org.apache.hadoop.fs.shell.find.TestFind   org.apache.hadoop.fs.shell.TestCopy   org.apache.hadoop.fs.shell.TestCount   org.apache.hadoop.fs.shell.TestCopyPreserveFlag   org.apache.hadoop.fs.shell.TestXAttrCommands   org.apache.hadoop.fs.TestFsShell   org.apache.hadoop.fs.TestLocalFSFileContextCreateMkdir   org.apache.hadoop.fs.TestSymlinkLocalFSFileContext   org.apache.hadoop.fs.TestS3_LocalFileContextURI   org.apache.hadoop.fs.TestFileSystemTokens   org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractAppend   org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractCreate   org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractOpen   org.apache.hadoop.fs.contract.localfs.TestLocalFSContractRename   org.apache.hadoop.fs.contract.localfs.TestLocalFSContractAppend   org.apache.hadoop.fs.contract.localfs.TestLocalFSContractCreate   org.apache.hadoop.fs.contract.ftp.TestFTPContractRename   org.apache.hadoop.fs.contract.ftp.TestFTPContractOpen   org.apache.hadoop.fs.contract.ftp.TestFTPContractMkdir   org.apache.hadoop.fs.contract.ftp.TestFTPContractCreate   org.apache.hadoop.fs.TestLocalFSFileContextMainOperations   org.apache.hadoop.fs.permission.TestFsPermission   org.apache.hadoop.fs.TestHarFileSystemBasics   org.apache.hadoop.fs.TestStat   org.apache.hadoop.fs.TestPath   org.apache.hadoop.util.TestHttpExceptionUtils   org.apache.hadoop.util.TestHostsFileReader   org.apache.hadoop.util.TestChunkedArrayList   org.apache.hadoop.util.TestLightWeightGSet   org.apache.hadoop.util.TestGenericOptionsParser   org.apache.hadoop.util.TestNativeCodeLoader   org.apache.hadoop.util.TestNativeLibraryChecker   org.apache.hadoop.util.TestLightWeightCache   org.apache.hadoop.util.TestWinUtils   org.apache.hadoop.util.TestProtoUtil   org.apache.hadoop.util.TestDirectBufferPool   org.apache.hadoop.util.TestGSet   org.apache.hadoop.http.TestPathFilter   org.apache.hadoop.http.TestHttpServer   org.apache.hadoop.http.TestHttpServerWebapps   org.apache.hadoop.crypto.TestCryptoCodec   org.apache.hadoop.crypto.key.TestKeyShell   org.apache.hadoop.io.TestWritableUtils   org.apache.hadoop.io.TestSequenceFileSync   org.apache.hadoop.io.TestWritable   org.apache.hadoop.io.TestEnumSetWritable   org.apache.hadoop.io.TestSequenceFile   org.apache.hadoop.io.TestTextNonUTF8   org.apache.hadoop.io.TestDefaultStringifier   org.apache.hadoop.io.TestArrayPrimitiveWritable   org.apache.hadoop.io.file.tfile.TestTFileComparators   org.apache.hadoop.io.file.tfile.TestTFileNoneCodecsByteArrays   org.apache.hadoop.io.file.tfile.TestTFileByteArrays   org.apache.hadoop.io.file.tfile.TestTFileLzoCodecsStreams   org.apache.hadoop.io.file.tfile.TestTFileComparator2   org.apache.hadoop.conf.TestReconfiguration   org.apache.hadoop.log.TestLog4Json   org.apache.hadoop.TestGenericRefresh   org.apache.hadoop.TestRefreshCallQueue   org.apache.hadoop.cli.TestCryptoAdminCLI   org.apache.hadoop.cli.TestHDFSCLI   org.apache.hadoop.cli.TestXAttrCLI   org.apache.hadoop.fs.viewfs.TestViewFileSystemWithXAttrs   org.apache.hadoop.fs.viewfs.TestViewFileSystemWithAcls   org.apache.hadoop.fs.shell.TestHdfsTextCommand   org.apache.hadoop.fs.contract.hdfs.TestHDFSContractSeek   org.apache.hadoop.fs.loadGenerator.TestLoadGenerator   org.apache.hadoop.fs.TestSymlinkHdfsFileSystem   org.apache.hadoop.tools.TestJMXGet   org.apache.hadoop.hdfs.TestParallelShortCircuitReadUnCached   org.apache.hadoop.hdfs.qjournal.server.TestJournalNodeMXBean   org.apache.hadoop.hdfs.qjournal.client.TestSegmentRecoveryComparator   org.apache.hadoop.hdfs.qjournal.client.TestQuorumCall   org.apache.hadoop.hdfs.TestWriteRead   org.apache.hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN   org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool   org.apache.hadoop.hdfs.server.datanode.TestStartSecureDataNode   org.apache.hadoop.hdfs.server.namenode.TestFileLimit   org.apache.hadoop.hdfs.server.namenode.TestFSImageWithSnapshot   org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandby   org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogsDuringFailover   org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   org.apache.hadoop.hdfs.server.namenode.TestCheckpoint   org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics   org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters   org.apache.hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics   org.apache.hadoop.hdfs.server.namenode.TestEditLogFileInputStream   org.apache.hadoop.hdfs.server.namenode.TestSecondaryWebUi   org.apache.hadoop.hdfs.server.balancer.TestBalancer   org.apache.hadoop.hdfs.web.TestWebHdfsWithAuthenticationFilter   org.apache.hadoop.hdfs.TestFsShellPermission   org.apache.hadoop.hdfs.TestRollingUpgrade   org.apache.hadoop.hdfs.TestParallelShortCircuitReadNoChecksum   org.apache.hadoop.hdfs.TestDFSRollback   org.apache.hadoop.hdfs.tools.TestDFSHAAdmin   org.apache.hadoop.hdfs.tools.TestDFSHAAdminMiniCluster   org.apache.hadoop.hdfs.TestBlockStoragePolicy   org.apache.hadoop.hdfs.TestCrcCorruption   org.apache.hadoop.hdfs.TestWriteConfigurationToDFS   org.apache.hadoop.hdfs.TestDataTransferKeepalive   org.apache.hadoop.hdfs.TestDFSStorageStateRecovery   org.apache.hadoop.hdfs.TestSnapshotCommands   org.apache.hadoop.hdfs.TestRollingUpgradeDowngrade   org.apache.hadoop.hdfs.TestGetBlocks   org.apache.hadoop.hdfs.TestReplication Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733351/HADOOP-11984.004.patch Optional Tests shellcheck javadoc javac unit git revision trunk / e453989 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6716/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6716/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6716/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6716/console This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          Please correct me if I'm wrong – Looks like that test-patch.sh relies on a lot of regex to do the job. Does it make sense to move some of these work towards a Python script so that it is possible to leverage more libraries?

          No, for the exact same reason the hadoop user-facing shell code isn't written in python.

          For example, To determine whether a test run successfully, it seems more reliable to parse the xml output of test result rather than doing a regex matching.

          The code in question:

              module_failed_tests=$(find . -name 'TEST*.xml'\
                | xargs "${GREP}" -l -E "<failure|<error"\
                | ${AWK} -F/ '{sub("TEST-org.apache.",""); sub(".xml",""); print $NF}')
          

          has been in use for about as long as test-patch as existed (minus some code cleanup and formatting changes by me). I doubt it's any less reliable than reading the XML directly since the only way this should break is if JUnit's XML DTD changes.... which, in turn, would break any replacement of this code as well. So it's a wash. (I'd also argue that this is likely going to be faster and might even use less memory, given how highly optimized grep implementations tend to be... but the awk might slow it down.)

          Show
          aw Allen Wittenauer added a comment - Please correct me if I'm wrong – Looks like that test-patch.sh relies on a lot of regex to do the job. Does it make sense to move some of these work towards a Python script so that it is possible to leverage more libraries? No, for the exact same reason the hadoop user-facing shell code isn't written in python. For example, To determine whether a test run successfully, it seems more reliable to parse the xml output of test result rather than doing a regex matching. The code in question: module_failed_tests=$(find . -name 'TEST*.xml'\ | xargs "${GREP}" -l -E "<failure|<error" \ | ${AWK} -F/ '{sub( "TEST-org.apache." , ""); sub(" .xml "," "); print $NF}') has been in use for about as long as test-patch as existed (minus some code cleanup and formatting changes by me). I doubt it's any less reliable than reading the XML directly since the only way this should break is if JUnit's XML DTD changes.... which, in turn, would break any replacement of this code as well. So it's a wash. (I'd also argue that this is likely going to be faster and might even use less memory, given how highly optimized grep implementations tend to be... but the awk might slow it down.)
          Hide
          wheat9 Haohui Mai added a comment -

          I doubt it's any less reliable than reading the XML directly since the only way this should break is if JUnit's XML DTD changes.... which, in turn, would break any replacement of this code as well.

          I'm unsure how the reliability / robustness argument holds. Just a quick skim it looks like that the code snippet assumes that JUnit pretty prints the XML results (i.e., the result of each test in a different line). Any XML parsers in production (e.g., xml.sax in Python) should handle non-pretty-printed XML well.

          I'd also argue that this is likely going to be faster and might even use less memory, given how highly optimized grep implementations tend to be... but the awk might slow it down.

          It seems to me that it is more of an apple vs orange comparison – more importantly, does the time parsing TEST-*xml (which takes seconds at maximum) actually matter, give the fact that in general Jenkins spends 15 mins to build the trunk, and ~2 hours to run the tests?

          Popping up one level – it looks like you have some concerns on moving test-patch to other scripting languages that have more choices of libraries. I understand you're probably have expertise in bash / awk but can you elaborate? test-patch.sh has been a minefield for a while and we're continuously adding new features into it.

          I appreciate your contributions on these scripts. It helps get rid a lot of headaches. I'm wondering whether there are anythings can be done to improve the maintainability and reduce the bars of getting involved (e.g., reusing libraries from other scripting languages) in the longer term.

          Show
          wheat9 Haohui Mai added a comment - I doubt it's any less reliable than reading the XML directly since the only way this should break is if JUnit's XML DTD changes.... which, in turn, would break any replacement of this code as well. I'm unsure how the reliability / robustness argument holds. Just a quick skim it looks like that the code snippet assumes that JUnit pretty prints the XML results (i.e., the result of each test in a different line). Any XML parsers in production (e.g., xml.sax in Python) should handle non-pretty-printed XML well. I'd also argue that this is likely going to be faster and might even use less memory, given how highly optimized grep implementations tend to be... but the awk might slow it down. It seems to me that it is more of an apple vs orange comparison – more importantly, does the time parsing TEST-*xml (which takes seconds at maximum) actually matter, give the fact that in general Jenkins spends 15 mins to build the trunk, and ~2 hours to run the tests? Popping up one level – it looks like you have some concerns on moving test-patch to other scripting languages that have more choices of libraries. I understand you're probably have expertise in bash / awk but can you elaborate? test-patch.sh has been a minefield for a while and we're continuously adding new features into it. I appreciate your contributions on these scripts. It helps get rid a lot of headaches. I'm wondering whether there are anythings can be done to improve the maintainability and reduce the bars of getting involved (e.g., reusing libraries from other scripting languages) in the longer term.
          Hide
          aw Allen Wittenauer added a comment -

          It seems to me that it is more of an apple vs orange comparison – more importantly, does the time parsing TEST-*xml (which takes seconds at maximum) actually matter, give the fact that in general Jenkins spends 15 mins to build the trunk, and ~2 hours to run the tests?

          ~2 hours only for HDFS. The next closest (IIRC) is mapreduce-jobclient which comes in at 20 minutes. Perhaps the HDFS folks should take a serious look at re-arranging the universe, not running integration tests in unit tests, start paying attention to the nightly build, etc.

          Popping up one level – it looks like you have some concerns on moving test-patch to other scripting languages that have more choices of libraries.

          deadhorse.gif

          Python, ruby, etc, all suffer from the same problem: which version do you target to get the maximum amount of coverage? test-patch, like the user-client code, MUST be able to run in a variety of hostile environments. (No, Mac OS X and Linux are NOT good enough.) python, frankly, sucks at that because the API is continually evolving in incompatible ways. ... and that's before we even get into the morass of add-ons. And python 3.x.

          FWIW, the only big portability problem with the current version of test-patch.sh that I'm aware of is one usage of GNU diff because I was too lazy to write more complex awk to work around it. Otherwise, it's all POSIX+bash 3.x and should run even on fairly ancient systems unchanged! The outlook for forward compatibility, as a result, is extremely good. It's pretty much impossible to do that with most other language choices (including, ironically, Java).... except maybe one:

          If I had my way, I'd have written this in perl 5. It's a significantly better choice for the things we need to do here (text processing! OS manipulation!) and it's compatibility across versions deployed with every relatively modern OS that I'm aware of is extremely high. But we don't do perl, have a small tolerance for python, and the rest is in bash. So given those choices, it was an easy one to make.

          I'm wondering whether there are anythings can be done to improve the maintainability and reduce the bars of getting involved (e.g., reusing libraries from other scripting languages) in the longer term.

          There are plenty of people who are fully competent to write decent bash. We just don't invite them into the Hadoop tent. The number of people contributing to the parts that I've rewritten have gone up SIGNIFICANTLY because people who have these skills realize that someone is paying attention. As a side note, I personally think it's great if the Java folks feel uncomfortable that code that they don't understand is in the system.

          - while working on releasedocmaker, I heard two conflicting things: "that API is deprecated you should use xyz" and "oh, make sure this works with python vx.x". Guess what? I can't use the non-deprecated API in vx.x. So deprecated APIs here we come, which now means I'm continually answering the question of "why does this code use method y?".

          Show
          aw Allen Wittenauer added a comment - It seems to me that it is more of an apple vs orange comparison – more importantly, does the time parsing TEST-*xml (which takes seconds at maximum) actually matter, give the fact that in general Jenkins spends 15 mins to build the trunk, and ~2 hours to run the tests? ~2 hours only for HDFS. The next closest (IIRC) is mapreduce-jobclient which comes in at 20 minutes. Perhaps the HDFS folks should take a serious look at re-arranging the universe, not running integration tests in unit tests, start paying attention to the nightly build, etc. Popping up one level – it looks like you have some concerns on moving test-patch to other scripting languages that have more choices of libraries. deadhorse.gif Python, ruby, etc, all suffer from the same problem: which version do you target to get the maximum amount of coverage? test-patch, like the user-client code, MUST be able to run in a variety of hostile environments. (No, Mac OS X and Linux are NOT good enough.) python, frankly, sucks at that because the API is continually evolving in incompatible ways. ... and that's before we even get into the morass of add-ons. And python 3.x. FWIW, the only big portability problem with the current version of test-patch.sh that I'm aware of is one usage of GNU diff because I was too lazy to write more complex awk to work around it. Otherwise, it's all POSIX+bash 3.x and should run even on fairly ancient systems unchanged! The outlook for forward compatibility, as a result, is extremely good. It's pretty much impossible to do that with most other language choices (including, ironically, Java).... except maybe one: If I had my way, I'd have written this in perl 5. It's a significantly better choice for the things we need to do here (text processing! OS manipulation!) and it's compatibility across versions deployed with every relatively modern OS that I'm aware of is extremely high. But we don't do perl, have a small tolerance for python, and the rest is in bash. So given those choices, it was an easy one to make. I'm wondering whether there are anythings can be done to improve the maintainability and reduce the bars of getting involved (e.g., reusing libraries from other scripting languages) in the longer term. There are plenty of people who are fully competent to write decent bash. We just don't invite them into the Hadoop tent. The number of people contributing to the parts that I've rewritten have gone up SIGNIFICANTLY because people who have these skills realize that someone is paying attention. As a side note, I personally think it's great if the Java folks feel uncomfortable that code that they don't understand is in the system. - while working on releasedocmaker, I heard two conflicting things: "that API is deprecated you should use xyz" and "oh, make sure this works with python vx.x". Guess what? I can't use the non-deprecated API in vx.x. So deprecated APIs here we come, which now means I'm continually answering the question of "why does this code use method y?".
          Hide
          cnauroth Chris Nauroth added a comment -

          Here is patch v005 with a fix for the timed-out test detection. It's still awk. We just need to use an associative array instead of line-by-line comparison.

          Show
          cnauroth Chris Nauroth added a comment - Here is patch v005 with a fix for the timed-out test detection. It's still awk. We just need to use an associative array instead of line-by-line comparison.
          Hide
          wheat9 Haohui Mai added a comment -

          ~2 hours only for HDFS. The next closest (IIRC) is mapreduce-jobclient which comes in at 20 minutes. Perhaps the HDFS folks should take a serious look at re-arranging the universe, not running integration tests in unit tests, start paying attention to the nightly build, etc.

          Obviously we can do better here and there, but even optimizing for the mapreduce-jobclient you're optimizing for 1% of the execution time. Is it worthwhile to trade the additional complexity and possibly reduced robustness for the optimized execution time?

          test-patch, ... MUST be able to run in a variety of hostile environments. (No, Mac OS X and Linux are NOT good enough.)

          Absolutely agree.

          FWIW, the only big portability problem with the current version of test-patch.sh

          The UNIX world supports bash pretty well. However, Hadoop officially supports Windows as well. How does the compatibility story work for Windows folks? Unfortunately things like path separators are quite annoying to handle.

          If I had my way, I'd have written this in perl 5. It's a significantly better choice for the things we need to do here (text processing! OS manipulation!)

          To clarify I have no problems writing things in bash today. What I'm trying to explore here from a maintainability / productivity prospective whether we should allow writing supplement scripts in languages that have a wide range of libraries to take care of things like text processing / OS manipulation, regardless it's Python, Perl or other languages. From a longer term prospective, however, having the option to leverage libraries can help improve the productivity.

          Show
          wheat9 Haohui Mai added a comment - ~2 hours only for HDFS. The next closest (IIRC) is mapreduce-jobclient which comes in at 20 minutes. Perhaps the HDFS folks should take a serious look at re-arranging the universe, not running integration tests in unit tests, start paying attention to the nightly build, etc. Obviously we can do better here and there, but even optimizing for the mapreduce-jobclient you're optimizing for 1% of the execution time. Is it worthwhile to trade the additional complexity and possibly reduced robustness for the optimized execution time? test-patch, ... MUST be able to run in a variety of hostile environments. (No, Mac OS X and Linux are NOT good enough.) Absolutely agree. FWIW, the only big portability problem with the current version of test-patch.sh The UNIX world supports bash pretty well. However, Hadoop officially supports Windows as well. How does the compatibility story work for Windows folks? Unfortunately things like path separators are quite annoying to handle. If I had my way, I'd have written this in perl 5. It's a significantly better choice for the things we need to do here (text processing! OS manipulation!) To clarify I have no problems writing things in bash today. What I'm trying to explore here from a maintainability / productivity prospective whether we should allow writing supplement scripts in languages that have a wide range of libraries to take care of things like text processing / OS manipulation, regardless it's Python, Perl or other languages. From a longer term prospective, however, having the option to leverage libraries can help improve the productivity.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to test-patch or smart-apply-patch has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6732/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6732/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 14m 40s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 30s There were no new javac warning messages.
          +1 javadoc 9m 34s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          -1 common tests 19m 33s Tests failed in hadoop-common.
          -1 hdfs tests 44m 6s Tests failed in hadoop-hdfs.
              98m 5s  



          Reason Tests
          Failed unit tests hadoop.ha.TestZKFailoverController
            hadoop.ha.TestActiveStandbyElectorRealZK
            hadoop.security.TestCredentials
            hadoop.hdfs.TestLeaseRecovery2
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
          Timed out tests org.apache.hadoop.conf.TestConfiguration



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12733607/HADOOP-11984.005.patch
          Optional Tests shellcheck javadoc javac unit
          git revision trunk / cdfae44
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6732/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6732/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6732/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6732/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 14m 40s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 34s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 common tests 19m 33s Tests failed in hadoop-common. -1 hdfs tests 44m 6s Tests failed in hadoop-hdfs.     98m 5s   Reason Tests Failed unit tests hadoop.ha.TestZKFailoverController   hadoop.ha.TestActiveStandbyElectorRealZK   hadoop.security.TestCredentials   hadoop.hdfs.TestLeaseRecovery2   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.qjournal.TestSecureNNWithQJM Timed out tests org.apache.hadoop.conf.TestConfiguration Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733607/HADOOP-11984.005.patch Optional Tests shellcheck javadoc javac unit git revision trunk / cdfae44 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6732/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6732/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6732/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6732/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Some of the tests related to ZooKeeper are hitting bind exceptions. Our ClientBaseWithFixes is a fork of a ZooKeeper class, which calls ZooKeeper's PortAssignment. That ZooKeeper class is not capable of guaranteeing uniqueness of ports across multiple concurrent processes. I recently fixed this in ZOOKEEPER-2183 as part of speeding up their CI. I might need to explore some kind of partial port of those changes over here into Hadoop.

          I can't repro the OutOfMemoryError problems after setting reuseForks to false. I don't know if I'm just overloading these hosts by running multiple processes with a maximum 1 GB heap. I don't have much visibility into these hosts. If anyone else out there knows how to find host-level details, please let me know. Otherwise, I'll probably need to coordinate with infra.

          Show
          cnauroth Chris Nauroth added a comment - Some of the tests related to ZooKeeper are hitting bind exceptions. Our ClientBaseWithFixes is a fork of a ZooKeeper class, which calls ZooKeeper's PortAssignment . That ZooKeeper class is not capable of guaranteeing uniqueness of ports across multiple concurrent processes. I recently fixed this in ZOOKEEPER-2183 as part of speeding up their CI. I might need to explore some kind of partial port of those changes over here into Hadoop. I can't repro the OutOfMemoryError problems after setting reuseForks to false. I don't know if I'm just overloading these hosts by running multiple processes with a maximum 1 GB heap. I don't have much visibility into these hosts. If anyone else out there knows how to find host-level details, please let me know. Otherwise, I'll probably need to coordinate with infra.
          Hide
          aw Allen Wittenauer added a comment -

          Is it worthwhile to trade the additional complexity and possibly reduced robustness for the optimized execution time?

          I don't find the current code complex at all compared to some of the gymnastics happening in hadoop-functions.sh. I might have a different opinion if the docker bits ever get committed.

          The UNIX world supports bash pretty well. However, Hadoop officially supports Windows as well. How does the compatibility story work for Windows folks? Unfortunately things like path separators are quite annoying to handle.

          We've already done it for the client code in trunk. It's likely better than the DOS batch code. My hunch is that it won't be that bad to get test-patch.sh running under cygwin. More complex than the client code though.

          What I'm trying to explore here from a maintainability / productivity prospective whether we should allow writing supplement scripts in languages that have a wide range of libraries to take care of things like text processing / OS manipulation, regardless it's Python, Perl or other languages.

          Until the portability issue gets dealt with (which will happen as more time passes), the answer here is likely no without potentially significantly increasing the burden of the installed software set.

          Of course, unless someone is actually going to do the work, this whole conversation is sort of moot. Traditionally, the non-java parts of Hadoop are regularly ignored until it impacts someone's corporate goals. I don't expect that to change any time soon.

          I can't repro the OutOfMemoryError problems after setting reuseForks to false. I don't know if I'm just overloading these hosts by running multiple processes with a maximum 1 GB heap. I don't have much visibility into these hosts. If anyone else out there knows how to find host-level details, please let me know. Otherwise, I'll probably need to coordinate with infra.

          Hmm. I wonder if the fact that MAVEN_OPTS is set to "-Xmx3072m -XX:MaxPermSize=768m" prior to test-patch.sh invocation in the HADOOP precommit is part of the problem here.

          This is where it'd be super useful to run precommit in the Docker container so we could see if it was actually a maven version problem.

          Show
          aw Allen Wittenauer added a comment - Is it worthwhile to trade the additional complexity and possibly reduced robustness for the optimized execution time? I don't find the current code complex at all compared to some of the gymnastics happening in hadoop-functions.sh. I might have a different opinion if the docker bits ever get committed. The UNIX world supports bash pretty well. However, Hadoop officially supports Windows as well. How does the compatibility story work for Windows folks? Unfortunately things like path separators are quite annoying to handle. We've already done it for the client code in trunk. It's likely better than the DOS batch code. My hunch is that it won't be that bad to get test-patch.sh running under cygwin. More complex than the client code though. What I'm trying to explore here from a maintainability / productivity prospective whether we should allow writing supplement scripts in languages that have a wide range of libraries to take care of things like text processing / OS manipulation, regardless it's Python, Perl or other languages. Until the portability issue gets dealt with (which will happen as more time passes), the answer here is likely no without potentially significantly increasing the burden of the installed software set. Of course, unless someone is actually going to do the work, this whole conversation is sort of moot. Traditionally, the non-java parts of Hadoop are regularly ignored until it impacts someone's corporate goals. I don't expect that to change any time soon. I can't repro the OutOfMemoryError problems after setting reuseForks to false. I don't know if I'm just overloading these hosts by running multiple processes with a maximum 1 GB heap. I don't have much visibility into these hosts. If anyone else out there knows how to find host-level details, please let me know. Otherwise, I'll probably need to coordinate with infra. Hmm. I wonder if the fact that MAVEN_OPTS is set to "-Xmx3072m -XX:MaxPermSize=768m" prior to test-patch.sh invocation in the HADOOP precommit is part of the problem here. This is where it'd be super useful to run precommit in the Docker container so we could see if it was actually a maven version problem.
          Hide
          cnauroth Chris Nauroth added a comment -

          MAVEN_OPTS is set to "-Xmx3072m -XX:MaxPermSize=768m"

          Thanks for the tip. That could be a factor. In my environment, I set MAVEN_OPTS with a max 1 GB heap. The forked JUnit processes are set to use 1 GB max heap each. (This is coded into the pom.xml, so this part won't vary between build environments.) That means a pre-commit runs on Jenkins would be 3 GB + 4 * 1 GB = 7 GB total. If a significant amount of the host's RAM is already allocated to Jenkins and whatever else runs on the host, then maybe we're resource-constrained. I'll ask infra for verification.

          Show
          cnauroth Chris Nauroth added a comment - MAVEN_OPTS is set to "-Xmx3072m -XX:MaxPermSize=768m" Thanks for the tip. That could be a factor. In my environment, I set MAVEN_OPTS with a max 1 GB heap. The forked JUnit processes are set to use 1 GB max heap each. (This is coded into the pom.xml, so this part won't vary between build environments.) That means a pre-commit runs on Jenkins would be 3 GB + 4 * 1 GB = 7 GB total. If a significant amount of the host's RAM is already allocated to Jenkins and whatever else runs on the host, then maybe we're resource-constrained. I'll ask infra for verification.
          Hide
          aw Allen Wittenauer added a comment -

          We set the MAVEN_OPTS to "-Xms256m -Xmx512m" in the development Dockerfile. So I have zero confidence that the 3GB number represents what actually needs to be used. As I've been digging through the precommit checks, there appears to be a ton of bad design, bad decision making, or just plain old ignorance that's been hanging around for a very long time. This feels like one of them. I think it'd be worthwhile changing the precommit value, running some tests and see what falls out. I suspect the answer will be: everything's fine.

          Show
          aw Allen Wittenauer added a comment - We set the MAVEN_OPTS to "-Xms256m -Xmx512m" in the development Dockerfile. So I have zero confidence that the 3GB number represents what actually needs to be used. As I've been digging through the precommit checks, there appears to be a ton of bad design, bad decision making, or just plain old ignorance that's been hanging around for a very long time. This feels like one of them. I think it'd be worthwhile changing the precommit value, running some tests and see what falls out. I suspect the answer will be: everything's fine.
          Hide
          cnauroth Chris Nauroth added a comment -

          Allen Wittenauer, is there a way that I (or you) can get access to change the pre-commit code without taking an extra hop through infra? When I login to builds.apache.org, I don't get an "edit job" link, so it looks like I don't have the rights.

          There is a reason we went to 1 GB in MAVEN_OPTS, but I don't remember the reason. I expect anything higher than that it unnecessary, as long as we're talking about just Hadoop pre-commit and not any kind of shared code that interacts with other projects. I don't know the heap requirements for other projects.

          Show
          cnauroth Chris Nauroth added a comment - Allen Wittenauer , is there a way that I (or you) can get access to change the pre-commit code without taking an extra hop through infra? When I login to builds.apache.org, I don't get an "edit job" link, so it looks like I don't have the rights. There is a reason we went to 1 GB in MAVEN_OPTS, but I don't remember the reason. I expect anything higher than that it unnecessary, as long as we're talking about just Hadoop pre-commit and not any kind of shared code that interacts with other projects. I don't know the heap requirements for other projects.
          Hide
          aw Allen Wittenauer added a comment - - edited

          is there a way that I (or you) can get access to change the pre-commit code without taking an extra hop through infra?

          I've got access. Chris Douglas has the ability to add people. (Although I suspect all PMC does) I'll change HADOOP precommit to 1GB here in a sec and let's see what happens. I'll leave the other Hadoop sub projects alone for the time being.

          Show
          aw Allen Wittenauer added a comment - - edited is there a way that I (or you) can get access to change the pre-commit code without taking an extra hop through infra? I've got access. Chris Douglas has the ability to add people. (Although I suspect all PMC does) I'll change HADOOP precommit to 1GB here in a sec and let's see what happens. I'll leave the other Hadoop sub projects alone for the time being.
          Hide
          chris.douglas Chris Douglas added a comment -

          Chris Nauroth I added you to the hudson-jobadmin group, so you should have access.

          Show
          chris.douglas Chris Douglas added a comment - Chris Nauroth I added you to the hudson-jobadmin group, so you should have access.
          Hide
          aw Allen Wittenauer added a comment -

          Changed the maven flags and re-kicked off this issue.

          Show
          aw Allen Wittenauer added a comment - Changed the maven flags and re-kicked off this issue.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to test-patch or smart-apply-patch has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6741/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6741/console in case of problems.
          Hide
          cnauroth Chris Nauroth added a comment -

          I added you to the hudson-jobadmin group

          Awesome, I love power! Thanks, Chris Douglas and Allen Wittenauer.

          Show
          cnauroth Chris Nauroth added a comment - I added you to the hudson-jobadmin group Awesome, I love power! Thanks, Chris Douglas and Allen Wittenauer .
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 14m 38s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 35s There were no new javac warning messages.
          +1 javadoc 9m 36s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          -1 common tests 19m 42s Tests failed in hadoop-common.
          -1 hdfs tests 44m 44s Tests failed in hadoop-hdfs.
              98m 56s  



          Reason Tests
          Failed unit tests hadoop.ha.TestZKFailoverControllerStress
            hadoop.ha.TestZKFailoverController
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.TestAppendSnapshotTruncate
            hadoop.hdfs.TestLeaseRecovery2
          Timed out tests org.apache.hadoop.conf.TestConfiguration



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12733607/HADOOP-11984.005.patch
          Optional Tests shellcheck javadoc javac unit
          git revision trunk / 8860e35
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6741/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6741/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6741/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6741/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 14m 38s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 35s There were no new javac warning messages. +1 javadoc 9m 36s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 common tests 19m 42s Tests failed in hadoop-common. -1 hdfs tests 44m 44s Tests failed in hadoop-hdfs.     98m 56s   Reason Tests Failed unit tests hadoop.ha.TestZKFailoverControllerStress   hadoop.ha.TestZKFailoverController   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.TestLeaseRecovery2 Timed out tests org.apache.hadoop.conf.TestConfiguration Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733607/HADOOP-11984.005.patch Optional Tests shellcheck javadoc javac unit git revision trunk / 8860e35 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6741/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6741/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6741/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6741/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          I finally was able to repro some of these OutOfMemoryError conditions locally.

          I figured out that HADOOP-11363 introduced configurability of the max heap size for forked Surefire processes. As part of that, it also increased the default setting from 1 GB to 4 GB, but only in the case where the build was not called with -Pparallel-tests. 4 GB is pretty high, and the comments in HADOOP-11363 suggest that it was going well beyond what was really needed. I'm going to try downtuning this to 2 GB and making sure the same value gets used for -Pparallel-tests.

          We also have an extremely naughty test, TestConfiguration#testConcurrentAccesses, that was introduced in HADOOP-11209. It works by spinning up 100 threads, each of which tries to put 100000 key-value pairs into a Configuration. When I run this beast with anything less than 4 GB, it goes into heavy GC churn before the threads can complete. That's why it's been reported as a timeout in the last several runs. (BTW, that's good validation that my change to the timed out test detection awk script actually works.) I'm temporarily removing this test to unblock the current testing effort while I think about a permanent fix.

          I'm uploading patch v006.

          Show
          cnauroth Chris Nauroth added a comment - I finally was able to repro some of these OutOfMemoryError conditions locally. I figured out that HADOOP-11363 introduced configurability of the max heap size for forked Surefire processes. As part of that, it also increased the default setting from 1 GB to 4 GB, but only in the case where the build was not called with -Pparallel-tests . 4 GB is pretty high, and the comments in HADOOP-11363 suggest that it was going well beyond what was really needed. I'm going to try downtuning this to 2 GB and making sure the same value gets used for -Pparallel-tests . We also have an extremely naughty test, TestConfiguration#testConcurrentAccesses , that was introduced in HADOOP-11209 . It works by spinning up 100 threads, each of which tries to put 100000 key-value pairs into a Configuration . When I run this beast with anything less than 4 GB, it goes into heavy GC churn before the threads can complete. That's why it's been reported as a timeout in the last several runs. (BTW, that's good validation that my change to the timed out test detection awk script actually works.) I'm temporarily removing this test to unblock the current testing effort while I think about a permanent fix. I'm uploading patch v006.
          Hide
          aw Allen Wittenauer added a comment -

          Keep in mind that precommit runs with Java 7 but HADOOP-11363 hit the problems with Java 8...

          Show
          aw Allen Wittenauer added a comment - Keep in mind that precommit runs with Java 7 but HADOOP-11363 hit the problems with Java 8...
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to test-patch or smart-apply-patch has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6755/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6755/console in case of problems.
          Hide
          cnauroth Chris Nauroth added a comment -

          Eventually, we saw the same problems on Java 7 too. I believe an earlier version of the patch that used 2 GB would have addressed it. Still, this is something that goes into my list of follow-ups before we can commit a patch here. I'll involve the original authors of that patch.

          Show
          cnauroth Chris Nauroth added a comment - Eventually, we saw the same problems on Java 7 too. I believe an earlier version of the patch that used 2 GB would have addressed it. Still, this is something that goes into my list of follow-ups before we can commit a patch here. I'll involve the original authors of that patch.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 15m 20s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 55s There were no new javac warning messages.
          +1 javadoc 10m 10s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 3m 36s There were no new checkstyle issues.
          +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 4m 46s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 7m 26s Tests failed in hadoop-common.
          -1 hdfs tests 41m 56s Tests failed in hadoop-hdfs.
              93m 51s  



          Reason Tests
          Failed unit tests hadoop.ha.TestZKFailoverController
            hadoop.ha.TestActiveStandbyElectorRealZK
            hadoop.hdfs.server.namenode.TestNameNodeHttpServer



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12733987/HADOOP-11984.006.patch
          Optional Tests shellcheck javadoc javac unit findbugs checkstyle
          git revision trunk / 12d6c5c
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6755/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6755/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6755/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6755/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 15m 20s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 55s There were no new javac warning messages. +1 javadoc 10m 10s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 3m 36s There were no new checkstyle issues. +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 4m 46s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 7m 26s Tests failed in hadoop-common. -1 hdfs tests 41m 56s Tests failed in hadoop-hdfs.     93m 51s   Reason Tests Failed unit tests hadoop.ha.TestZKFailoverController   hadoop.ha.TestActiveStandbyElectorRealZK   hadoop.hdfs.server.namenode.TestNameNodeHttpServer Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733987/HADOOP-11984.006.patch Optional Tests shellcheck javadoc javac unit findbugs checkstyle git revision trunk / 12d6c5c hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6755/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6755/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6755/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6755/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          I'm uploading patch v007. Here are the differences since last time:

          1. I'm no longer refactoring the parallel-tests profile up to hadoop-project/pom.xml. YARN and MapReduce aren't ready to inherit this, so it will have to be duplicated in hadoop-common and hadoop-hdfs for now. We can refactor later after the corresponding work is done in YARN and MapReduce.
          2. TestConfiguration#testConcurrentAccesses is no longer being removed, but I did lower its footprint by running fewer iterations in the main loop of each thread. I verified that this is still sufficient to catch the bug it was written for by temporarily reverting the Configuration fix that went with this test.
          3. Some tests use test.build.dir instead of test.build.data, so I set that up to be parameterized by the JUnit process number.
          4. To resolve the ZooKeeper-related failures, I introduced test.build.shared.data, which is guaranteed to be shared across all of the forked JUnit processes. ClientBaseWithFixes had already been set up to use file locking to prevent port sharing, but since test.build.data is parameterized by JUnit process ID now, each process would take its lock files in a separate directory. That defeats the purpose of file locking. By moving this back to a shared directory, the file locking works again.
          5. KeyStoreTestUtil was using target/test-classes for putting SSL configuration files on the classpath. Concurrent tests could collide and mess up their configurations. Unfortunately, we can't use surefire.forkNumber in an additionalClasspathElement due to Maven limitations. Instead, I've set it up so that the individual SSL configuration file names can be parameterized by JUnit process number. Several tests have been updated to set their configurations accordingly.
          Show
          cnauroth Chris Nauroth added a comment - I'm uploading patch v007. Here are the differences since last time: I'm no longer refactoring the parallel-tests profile up to hadoop-project/pom.xml. YARN and MapReduce aren't ready to inherit this, so it will have to be duplicated in hadoop-common and hadoop-hdfs for now. We can refactor later after the corresponding work is done in YARN and MapReduce. TestConfiguration#testConcurrentAccesses is no longer being removed, but I did lower its footprint by running fewer iterations in the main loop of each thread. I verified that this is still sufficient to catch the bug it was written for by temporarily reverting the Configuration fix that went with this test. Some tests use test.build.dir instead of test.build.data , so I set that up to be parameterized by the JUnit process number. To resolve the ZooKeeper-related failures, I introduced test.build.shared.data , which is guaranteed to be shared across all of the forked JUnit processes. ClientBaseWithFixes had already been set up to use file locking to prevent port sharing, but since test.build.data is parameterized by JUnit process ID now, each process would take its lock files in a separate directory. That defeats the purpose of file locking. By moving this back to a shared directory, the file locking works again. KeyStoreTestUtil was using target/test-classes for putting SSL configuration files on the classpath. Concurrent tests could collide and mess up their configurations. Unfortunately, we can't use surefire.forkNumber in an additionalClasspathElement due to Maven limitations. Instead, I've set it up so that the individual SSL configuration file names can be parameterized by JUnit process number. Several tests have been updated to set their configurations accordingly.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to test-patch or smart-apply-patch has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6803/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6803/console in case of problems.
          Hide
          aw Allen Wittenauer added a comment -

          It would probably be good to have a flag to turn parallel tests on and off as well as one to set how many to run concurrently. This way folks running test-patch on their laptops aren't taken by surprised.

          Show
          aw Allen Wittenauer added a comment - It would probably be good to have a flag to turn parallel tests on and off as well as one to set how many to run concurrently. This way folks running test-patch on their laptops aren't taken by surprised.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 14m 44s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 tests included 0m 0s The patch appears to include 11 new or modified test files.
          +1 javac 7m 33s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 47s There were no new checkstyle issues.
          +1 shellcheck 0m 9s There were no new shellcheck (v0.3.3) issues.
          -1 whitespace 0m 2s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 39s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 43s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 7m 32s Tests failed in hadoop-common.
          -1 hdfs tests 43m 4s Tests failed in hadoop-hdfs.
              92m 52s  



          Reason Tests
          Failed unit tests hadoop.security.TestCredentials
            hadoop.hdfs.TestDFSClientRetries



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12734991/HADOOP-11984.007.patch
          Optional Tests shellcheck javadoc javac unit findbugs checkstyle
          git revision trunk / 446d515
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6803/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6803/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6803/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6803/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6803/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 14m 44s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 tests included 0m 0s The patch appears to include 11 new or modified test files. +1 javac 7m 33s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 47s There were no new checkstyle issues. +1 shellcheck 0m 9s There were no new shellcheck (v0.3.3) issues. -1 whitespace 0m 2s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 39s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 43s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 7m 32s Tests failed in hadoop-common. -1 hdfs tests 43m 4s Tests failed in hadoop-hdfs.     92m 52s   Reason Tests Failed unit tests hadoop.security.TestCredentials   hadoop.hdfs.TestDFSClientRetries Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12734991/HADOOP-11984.007.patch Optional Tests shellcheck javadoc javac unit findbugs checkstyle git revision trunk / 446d515 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6803/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6803/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6803/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6803/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6803/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          I'm really stumped about TestCredentials. I can't repro the failure locally. It seems like this kind of error can happen only if the directory creation during setUp failed. Patch v008 just adds some logging around that to try to help troubleshoot.

          Show
          cnauroth Chris Nauroth added a comment - I'm really stumped about TestCredentials . I can't repro the failure locally. It seems like this kind of error can happen only if the directory creation during setUp failed. Patch v008 just adds some logging around that to try to help troubleshoot.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to test-patch or smart-apply-patch has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6810/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6810/console in case of problems.
          Hide
          aw Allen Wittenauer added a comment -

          It seems like this kind of error can happen only if the directory creation during setUp failed.

          This reeks of a race condition with another test that is using the same value of test.build.data/mapred (why is it mapred anyway?) or perhaps even a naked test.build.data and nuking the contents out from underneath it.

          Show
          aw Allen Wittenauer added a comment - It seems like this kind of error can happen only if the directory creation during setUp failed. This reeks of a race condition with another test that is using the same value of test.build.data/mapred (why is it mapred anyway?) or perhaps even a naked test.build.data and nuking the contents out from underneath it.
          Hide
          cnauroth Chris Nauroth added a comment -

          Yes, I looked around for both of those things, but I didn't spot any problems. (That doesn't mean it doesn't exist, only that I didn't find anything.)

          Show
          cnauroth Chris Nauroth added a comment - Yes, I looked around for both of those things, but I didn't spot any problems. (That doesn't mean it doesn't exist, only that I didn't find anything.)
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 14m 34s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 tests included 0m 0s The patch appears to include 12 new or modified test files.
          +1 javac 7m 31s There were no new javac warning messages.
          +1 javadoc 9m 28s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 46s There were no new checkstyle issues.
          +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues.
          -1 whitespace 0m 1s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 4m 40s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 7m 38s Tests failed in hadoop-common.
          -1 hdfs tests 41m 56s Tests failed in hadoop-hdfs.
              91m 16s  



          Reason Tests
          Failed unit tests hadoop.security.TestCredentials
            hadoop.hdfs.server.namenode.TestCacheDirectives



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12735085/HADOOP-11984.008.patch
          Optional Tests shellcheck javadoc javac unit findbugs checkstyle
          git revision trunk / 446d515
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6810/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6810/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6810/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6810/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6810/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 14m 34s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 tests included 0m 0s The patch appears to include 12 new or modified test files. +1 javac 7m 31s There were no new javac warning messages. +1 javadoc 9m 28s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 46s There were no new checkstyle issues. +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues. -1 whitespace 0m 1s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 4m 40s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 7m 38s Tests failed in hadoop-common. -1 hdfs tests 41m 56s Tests failed in hadoop-hdfs.     91m 16s   Reason Tests Failed unit tests hadoop.security.TestCredentials   hadoop.hdfs.server.namenode.TestCacheDirectives Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735085/HADOOP-11984.008.patch Optional Tests shellcheck javadoc javac unit findbugs checkstyle git revision trunk / 446d515 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6810/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6810/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6810/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6810/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6810/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          As I suspected, the base directory wasn't in place at the time the test started.

          Patch v009 is another troubleshooting patch. The pom.xml has been changed to fail fast if the mkdir of the parallel testing directories fails. This will only run TestCredentials. This is going to help narrow down if the problem is happening at initial directory creation time or if it might be another test deleting the whole directory.

          Show
          cnauroth Chris Nauroth added a comment - As I suspected, the base directory wasn't in place at the time the test started. Patch v009 is another troubleshooting patch. The pom.xml has been changed to fail fast if the mkdir of the parallel testing directories fails. This will only run TestCredentials . This is going to help narrow down if the problem is happening at initial directory creation time or if it might be another test deleting the whole directory.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to test-patch or smart-apply-patch has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6826/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6826/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 14m 35s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 tests included 0m 0s The patch appears to include 6 new or modified test files.
          +1 javac 7m 29s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 6s There were no new checkstyle issues.
          +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues.
          -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 39s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 8m 9s Tests passed in hadoop-common.
              45m 22s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12735351/HADOOP-11984.009.patch
          Optional Tests shellcheck javadoc javac unit findbugs checkstyle
          git revision trunk / 022f49d
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6826/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6826/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6826/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6826/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 14m 35s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 tests included 0m 0s The patch appears to include 6 new or modified test files. +1 javac 7m 29s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 6s There were no new checkstyle issues. +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues. -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 39s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 8m 9s Tests passed in hadoop-common.     45m 22s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735351/HADOOP-11984.009.patch Optional Tests shellcheck javadoc javac unit findbugs checkstyle git revision trunk / 022f49d whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6826/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6826/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6826/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6826/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Patch v010 fixes a problem in the last experiment that I was trying.

          Show
          cnauroth Chris Nauroth added a comment - Patch v010 fixes a problem in the last experiment that I was trying.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to test-patch or smart-apply-patch has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6827/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6827/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 14m 36s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 tests included 0m 0s The patch appears to include 6 new or modified test files.
          +1 javac 7m 30s There were no new javac warning messages.
          +1 javadoc 9m 41s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 5s There were no new checkstyle issues.
          +1 shellcheck 0m 9s There were no new shellcheck (v0.3.3) issues.
          -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 39s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 1m 23s Tests passed in hadoop-common.
              38m 38s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12735365/HADOOP-11984.010.patch
          Optional Tests shellcheck javadoc javac unit findbugs checkstyle
          git revision trunk / 022f49d
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6827/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6827/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6827/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6827/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 14m 36s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 tests included 0m 0s The patch appears to include 6 new or modified test files. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 41s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 5s There were no new checkstyle issues. +1 shellcheck 0m 9s There were no new shellcheck (v0.3.3) issues. -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 39s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 1m 23s Tests passed in hadoop-common.     38m 38s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735365/HADOOP-11984.010.patch Optional Tests shellcheck javadoc javac unit findbugs checkstyle git revision trunk / 022f49d whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6827/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6827/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6827/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6827/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Even when TestCredentials runs in isolation, the parent directory isn't there, so that rules out another concurrent test interfering. This is very strange. Is something going wrong with the mkdir of the test directories inside pom.xml? I wouldn't expect so, because we'd see error output and an earlier failure in the build.

          The fix might be just to change TestCredentials to use a recursive mkdirs, which is what other tests do. I'm really curious about this though, so patch v011 is one more troubleshooting patch that echoes the directories that pom.xml tries to create. Let's see if these are any different from what I see on my local machine.

          Show
          cnauroth Chris Nauroth added a comment - Even when TestCredentials runs in isolation, the parent directory isn't there, so that rules out another concurrent test interfering. This is very strange. Is something going wrong with the mkdir of the test directories inside pom.xml? I wouldn't expect so, because we'd see error output and an earlier failure in the build. The fix might be just to change TestCredentials to use a recursive mkdirs , which is what other tests do. I'm really curious about this though, so patch v011 is one more troubleshooting patch that echoes the directories that pom.xml tries to create. Let's see if these are any different from what I see on my local machine.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to test-patch or smart-apply-patch has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6829/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6829/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 15m 20s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 tests included 0m 0s The patch appears to include 6 new or modified test files.
          +1 javac 7m 48s There were no new javac warning messages.
          +1 javadoc 10m 1s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 6s There were no new checkstyle issues.
          +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues.
          -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 43s The patch built with eclipse:eclipse.
          +1 findbugs 1m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 1m 16s Tests passed in hadoop-common.
              40m 16s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12735382/HADOOP-11984.011.patch
          Optional Tests shellcheck javadoc javac unit findbugs checkstyle
          git revision trunk / 10732d5
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6829/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6829/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6829/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6829/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 15m 20s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 tests included 0m 0s The patch appears to include 6 new or modified test files. +1 javac 7m 48s There were no new javac warning messages. +1 javadoc 10m 1s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 6s There were no new checkstyle issues. +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues. -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 43s The patch built with eclipse:eclipse. +1 findbugs 1m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 1m 16s Tests passed in hadoop-common.     40m 16s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735382/HADOOP-11984.011.patch Optional Tests shellcheck javadoc javac unit findbugs checkstyle git revision trunk / 10732d5 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6829/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6829/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6829/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6829/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Ah-ha! There is our root cause:

               [exec] Running mkdir test.build.data: /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/hadoop-common-project/hadoop-common/target/test/data/{1..4} test.build.dir: /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/hadoop-common-project/hadoop-common/target/test-dir/{1..4} hadoop.tmp.dir: /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/hadoop-common-project/hadoop-common/target/test/{1..4}
          

          It looks like Jenkins (and only Jenkins so far) is running a shell that isn't expanding {1..4} in the loop. I don't have enough historical shell knowledge here to know if only certain shells promise support for this syntax. I suppose I could change this to call seq, which is part of coreutils, but then I don't really know the portability promises on that one either.

          Allen Wittenauer, can I get some advice from you? What would be the most portable way to write the following kind of loop?

          for i in {1..4}; do mkdir -p myDirectory/$i; done
          
          Show
          cnauroth Chris Nauroth added a comment - Ah-ha! There is our root cause: [exec] Running mkdir test.build.data: /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/hadoop-common-project/hadoop-common/target/test/data/{1..4} test.build.dir: /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/hadoop-common-project/hadoop-common/target/test-dir/{1..4} hadoop.tmp.dir: /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/hadoop-common-project/hadoop-common/target/test/{1..4} It looks like Jenkins (and only Jenkins so far) is running a shell that isn't expanding { 1..4 } in the loop. I don't have enough historical shell knowledge here to know if only certain shells promise support for this syntax. I suppose I could change this to call seq , which is part of coreutils, but then I don't really know the portability promises on that one either. Allen Wittenauer , can I get some advice from you? What would be the most portable way to write the following kind of loop? for i in {1..4}; do mkdir -p myDirectory/$i; done
          Hide
          cnauroth Chris Nauroth added a comment -

          Maybe this is a good reason to bring out my old frenemy, Javascript. We can use the Ant <script> task to run Javascript through Rhino. This executes in the JDK, so it removes a dependency on any particular external shell.

          Here is patch v013 trying that approach. I also added the new arguments to test-patch.sh as requested by Allen.

          Show
          cnauroth Chris Nauroth added a comment - Maybe this is a good reason to bring out my old frenemy, Javascript. We can use the Ant <script> task to run Javascript through Rhino. This executes in the JDK, so it removes a dependency on any particular external shell. Here is patch v013 trying that approach. I also added the new arguments to test-patch.sh as requested by Allen.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to test-patch or smart-apply-patch has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6852/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6852/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 18m 6s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 tests included 0m 0s The patch appears to include 11 new or modified test files.
          +1 javac 8m 52s There were no new javac warning messages.
          +1 javadoc 10m 21s There were no new javadoc warning messages.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 58s There were no new checkstyle issues.
          +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues.
          -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 4m 42s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 8m 8s Tests passed in hadoop-common.
          -1 hdfs tests 42m 9s Tests failed in hadoop-hdfs.
              98m 1s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.namenode.ha.TestHAAppend



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12735801/HADOOP-11984.013.patch
          Optional Tests shellcheck javadoc javac unit findbugs checkstyle
          git revision trunk / 5450413
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6852/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6852/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6852/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6852/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6852/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 18m 6s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 tests included 0m 0s The patch appears to include 11 new or modified test files. +1 javac 8m 52s There were no new javac warning messages. +1 javadoc 10m 21s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 58s There were no new checkstyle issues. +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues. -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 4m 42s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 8m 8s Tests passed in hadoop-common. -1 hdfs tests 42m 9s Tests failed in hadoop-hdfs.     98m 1s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.ha.TestHAAppend Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735801/HADOOP-11984.013.patch Optional Tests shellcheck javadoc javac unit findbugs checkstyle git revision trunk / 5450413 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6852/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6852/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6852/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6852/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6852/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          This is awesome.
          As I checked, on same H3 machine only HDFS tests took 1hr 54 Min, least before.

          But now, Common+HDFS tests took just 98m.
          This is extra-ordinary work.
          Thanks Chris Nauroth.

          I haven't checked the patch completely though, just got attracted by the build time

          Show
          vinayrpet Vinayakumar B added a comment - This is awesome. As I checked, on same H3 machine only HDFS tests took 1hr 54 Min, least before. But now, Common+HDFS tests took just 98m. This is extra-ordinary work. Thanks Chris Nauroth . I haven't checked the patch completely though, just got attracted by the build time
          Hide
          cnauroth Chris Nauroth added a comment -

          Vinayakumar B, thank you!

          I just want to clarify something about those timings too. The 98 minutes that you mentioned was for the full pre-commit job run (checking javac warnings, javadoc warnings, checkstyle, etc.). If we look at just the test execution time, hadoop-common tests are down from ~24 minutes to ~8 minutes, and hadoop-hdfs tests are down from ~2 hours to ~45 minutes.

          For hadoop-common, we're approaching a situation where test execution is no longer the long pole in the overall pre-commit job execution time. There is still plenty of room for optimization in the HDFS tests though.

          Show
          cnauroth Chris Nauroth added a comment - Vinayakumar B , thank you! I just want to clarify something about those timings too. The 98 minutes that you mentioned was for the full pre-commit job run (checking javac warnings, javadoc warnings, checkstyle, etc.). If we look at just the test execution time, hadoop-common tests are down from ~24 minutes to ~8 minutes, and hadoop-hdfs tests are down from ~2 hours to ~45 minutes. For hadoop-common, we're approaching a situation where test execution is no longer the long pole in the overall pre-commit job execution time. There is still plenty of room for optimization in the HDFS tests though.
          Hide
          aw Allen Wittenauer added a comment -

          We need to try this patch on OS X. My test run jsut enabling parallel tests was... not good. After 12 hours, I had to kill it.

          Show
          aw Allen Wittenauer added a comment - We need to try this patch on OS X. My test run jsut enabling parallel tests was... not good. After 12 hours, I had to kill it.
          Hide
          cnauroth Chris Nauroth added a comment -

          I've been testing all revisions of this patch on both OS X and CentOS. I'll do Windows too as we get closer to committing. Before this patch, multiple devs were running the parallel-tests profile on OS X successfully, minus the couple of naughty tests I fixed here. Can you please try again and let me know if I can do anything to help troubleshoot?

          Show
          cnauroth Chris Nauroth added a comment - I've been testing all revisions of this patch on both OS X and CentOS. I'll do Windows too as we get closer to committing. Before this patch, multiple devs were running the parallel-tests profile on OS X successfully, minus the couple of naughty tests I fixed here. Can you please try again and let me know if I can do anything to help troubleshoot?
          Hide
          busbey Sean Busbey added a comment -

          given the variety, it would help if you both documented what version of OSX/maven and where your maven comes from (i.e. OSX vs homebrew)

          Show
          busbey Sean Busbey added a comment - given the variety, it would help if you both documented what version of OSX/maven and where your maven comes from (i.e. OSX vs homebrew)
          Hide
          cnauroth Chris Nauroth added a comment -

          Here is my info. I'm on OS X 10.9.5, my Java version is 1.7.0_67, and my Maven is version 3.0.5. For some reason, that doesn't match with the port package version, which is 3.2.3.

          > port provides $(which mvn)
          /opt/local/share/java/maven3/bin/mvn is provided by: maven3
          
          > port list maven3
          maven3                         @3.2.3          java/maven3
          
          > mvn --version
          Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 05:51:28-0800)
          Maven home: /opt/local/share/java/maven3
          Java version: 1.7.0_67, vendor: Oracle Corporation
          Java home: /Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/jre
          Default locale: en_US, platform encoding: UTF-8
          OS name: "mac os x", version: "10.9.5", arch: "x86_64", family: "mac"
          
          > sw_vers -productVersion
          10.9.5
          
          > uname -a
          Darwin Chriss-MacBook-Pro.local 13.4.0 Darwin Kernel Version 13.4.0: Wed Dec 17 19:05:52 PST 2014; root:xnu-2422.115.10~1/RELEASE_X86_64 x86_64
          
          Show
          cnauroth Chris Nauroth added a comment - Here is my info. I'm on OS X 10.9.5, my Java version is 1.7.0_67, and my Maven is version 3.0.5. For some reason, that doesn't match with the port package version, which is 3.2.3. > port provides $(which mvn) /opt/local/share/java/maven3/bin/mvn is provided by: maven3 > port list maven3 maven3 @3.2.3 java/maven3 > mvn --version Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 05:51:28-0800) Maven home: /opt/local/share/java/maven3 Java version: 1.7.0_67, vendor: Oracle Corporation Java home: /Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/jre Default locale: en_US, platform encoding: UTF-8 OS name: "mac os x" , version: "10.9.5" , arch: "x86_64" , family: "mac" > sw_vers -productVersion 10.9.5 > uname -a Darwin Chriss-MacBook-Pro.local 13.4.0 Darwin Kernel Version 13.4.0: Wed Dec 17 19:05:52 PST 2014; root:xnu-2422.115.10~1/RELEASE_X86_64 x86_64
          Hide
          cnauroth Chris Nauroth added a comment -

          Patch v014 adds a change to TestHAAppend to allow for a longer block recovery time, increasing from the default of 30 seconds to 60 seconds. In repeated runs locally, I'm seeing that there is a small chance that running concurrent tests puts enough extra load on the system that TestHAAppend doesn't make progress fast enough to satisfy its current timeout.

          Regrettably, many of our existing tests are based on particular timings, so it's not surprising that increasing system load exposes problems like this. Tuning the timings has been a perpetual whack-a-mole problem, and it isn't deterministic. I'd like us to improve on this as part of the HDFS-8381 work.

          Show
          cnauroth Chris Nauroth added a comment - Patch v014 adds a change to TestHAAppend to allow for a longer block recovery time, increasing from the default of 30 seconds to 60 seconds. In repeated runs locally, I'm seeing that there is a small chance that running concurrent tests puts enough extra load on the system that TestHAAppend doesn't make progress fast enough to satisfy its current timeout. Regrettably, many of our existing tests are based on particular timings, so it's not surprising that increasing system load exposes problems like this. Tuning the timings has been a perpetual whack-a-mole problem, and it isn't deterministic. I'd like us to improve on this as part of the HDFS-8381 work.
          Hide
          busbey Sean Busbey added a comment -

          my Maven is version 3.0.5. For some reason, that doesn't match with the port package version, which is 3.2.3.

          This part is probably because you're falling back to the version that comes with OS X. On my homebrew setup, this happens when /usr/local/bin isn't in the right place on my path. I'm guessing something similar for ports, just where ever it puts executables.

          Show
          busbey Sean Busbey added a comment - my Maven is version 3.0.5. For some reason, that doesn't match with the port package version, which is 3.2.3. This part is probably because you're falling back to the version that comes with OS X. On my homebrew setup, this happens when /usr/local/bin isn't in the right place on my path. I'm guessing something similar for ports, just where ever it puts executables.
          Hide
          busbey Sean Busbey added a comment -

          my Maven is version 3.0.5. For some reason, that doesn't match with the port package version, which is 3.2.3.

          This part is probably because you're falling back to the version that comes with OS X. On my homebrew setup, this happens when /usr/local/bin isn't in the right place on my path. I'm guessing something similar for ports, just where ever it puts executables.

          Show
          busbey Sean Busbey added a comment - my Maven is version 3.0.5. For some reason, that doesn't match with the port package version, which is 3.2.3. This part is probably because you're falling back to the version that comes with OS X. On my homebrew setup, this happens when /usr/local/bin isn't in the right place on my path. I'm guessing something similar for ports, just where ever it puts executables.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 19s The patch command could not apply the patch.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12735928/HADOOP-11984.014.patch
          Optional Tests shellcheck javadoc javac unit findbugs checkstyle
          git revision trunk / 7ebe80e
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6857/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 19s The patch command could not apply the patch. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735928/HADOOP-11984.014.patch Optional Tests shellcheck javadoc javac unit findbugs checkstyle git revision trunk / 7ebe80e Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6857/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Patch v015 is a rebase after another test-patch.sh change got committed earlier today.

          Show
          cnauroth Chris Nauroth added a comment - Patch v015 is a rebase after another test-patch.sh change got committed earlier today.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to test-patch or smart-apply-patch has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6859/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6859/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 19m 30s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 tests included 0m 0s The patch appears to include 12 new or modified test files.
          +1 javac 7m 36s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 46s There were no new checkstyle issues.
          +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues.
          -1 whitespace 0m 2s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 37s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          +1 findbugs 5m 5s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 6m 57s Tests passed in hadoop-common.
          -1 hdfs tests 41m 39s Tests failed in hadoop-hdfs.
              96m 0s  



          Reason Tests
          Failed unit tests hadoop.hdfs.web.TestWebHDFS



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12735975/HADOOP-11984.015.patch
          Optional Tests shellcheck javadoc javac unit findbugs checkstyle
          git revision trunk / 788bfa0
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6859/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6859/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6859/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6859/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6859/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 19m 30s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 tests included 0m 0s The patch appears to include 12 new or modified test files. +1 javac 7m 36s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 46s There were no new checkstyle issues. +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues. -1 whitespace 0m 2s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 5m 5s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 6m 57s Tests passed in hadoop-common. -1 hdfs tests 41m 39s Tests failed in hadoop-hdfs.     96m 0s   Reason Tests Failed unit tests hadoop.hdfs.web.TestWebHDFS Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735975/HADOOP-11984.015.patch Optional Tests shellcheck javadoc javac unit findbugs checkstyle git revision trunk / 788bfa0 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6859/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6859/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6859/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6859/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6859/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Patch v016 is a rebase.

          TestWebHDFS had a socket read timeout. This is something I've seen intermittently even before this patch. I haven't been able to repro locally.

          Show
          cnauroth Chris Nauroth added a comment - Patch v016 is a rebase. TestWebHDFS had a socket read timeout. This is something I've seen intermittently even before this patch. I haven't been able to repro locally.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to test-patch or smart-apply-patch has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6875/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6875/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 19m 16s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 tests included 0m 0s The patch appears to include 12 new or modified test files.
          +1 javac 7m 30s There were no new javac warning messages.
          +1 javadoc 9m 41s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 43s There were no new checkstyle issues.
          +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues.
          -1 whitespace 0m 2s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 38s mvn install still works.
          +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
          +1 findbugs 5m 1s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 8m 5s Tests passed in hadoop-common.
          +1 hdfs tests 42m 36s Tests passed in hadoop-hdfs.
              97m 44s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12736351/HADOOP-11984.016.patch
          Optional Tests shellcheck javadoc javac unit findbugs checkstyle
          git revision trunk / eb6bf91
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6875/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6875/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6875/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6875/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6875/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 19m 16s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 tests included 0m 0s The patch appears to include 12 new or modified test files. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 41s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 43s There were no new checkstyle issues. +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues. -1 whitespace 0m 2s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 38s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. +1 findbugs 5m 1s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 8m 5s Tests passed in hadoop-common. +1 hdfs tests 42m 36s Tests passed in hadoop-hdfs.     97m 44s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12736351/HADOOP-11984.016.patch Optional Tests shellcheck javadoc javac unit findbugs checkstyle git revision trunk / eb6bf91 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6875/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6875/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6875/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6875/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6875/console This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment - - edited

          1. Minor nit:

          (default defined in Hadoop build)
          

          s,Hadoop,${PROJECT_NAME},g

          2. I'm confused as to what the intent here is:

          +  echo "--test-threads=<int>   Number of tests to run in parallel (default defined in Hadoop build)"
          

          OK, --test-threads takes an int.

          +      --test-threads=*)
          +        TEST_THREADS=${i#*=}
          +      ;;
          

          Mhmm ok.

          +  if [[ ${TEST_PARALLEL} == "true" ]] ; then
          +    PARALLEL_TESTS_PROFILE=-Pparallel-tests
          +    if [[ -z ${TEST_THREADS:-} ]]; then
          +      TESTS_THREAD_COUNT=$TEST_THREADS
          +    fi
          +  fi
          

          OK, so we've got a count var we're probably going to finish setting up later.

          +    echo_and_redirect "${test_logfile}" "${MVN}" "${MAVEN_ARGS[@]}" clean install -fae ${NATIVE_PROFILE} ${REQUIRE_TEST_LIB_HADOOP} ${PARALLEL_TESTS_PROFILE} ${TESTS_THREAD_COUNT} 
          

          ... umm, the number is now naked on the maven command line.

          If I had to guess, you intended for TESTS_THREAD_COUNT="-Dblah=TEST_COUNT" rather than be a number down here on the maven command line.

          Show
          aw Allen Wittenauer added a comment - - edited 1. Minor nit: ( default defined in Hadoop build) s,Hadoop,${PROJECT_NAME},g 2. I'm confused as to what the intent here is: + echo "--test-threads=< int > Number of tests to run in parallel ( default defined in Hadoop build)" OK, --test-threads takes an int. + --test-threads=*) + TEST_THREADS=${i#*=} + ;; Mhmm ok. + if [[ ${TEST_PARALLEL} == " true " ]] ; then + PARALLEL_TESTS_PROFILE=-Pparallel-tests + if [[ -z ${TEST_THREADS:-} ]]; then + TESTS_THREAD_COUNT=$TEST_THREADS + fi + fi OK, so we've got a count var we're probably going to finish setting up later. + echo_and_redirect "${test_logfile}" "${MVN}" "${MAVEN_ARGS[@]}" clean install -fae ${NATIVE_PROFILE} ${REQUIRE_TEST_LIB_HADOOP} ${PARALLEL_TESTS_PROFILE} ${TESTS_THREAD_COUNT} ... umm, the number is now naked on the maven command line. If I had to guess, you intended for TESTS_THREAD_COUNT="-Dblah=TEST_COUNT" rather than be a number down here on the maven command line.
          Hide
          cnauroth Chris Nauroth added a comment -

          Thank you for the review, Allen. You're right about passing the parameter incorrectly to Maven. I think I introduced that bug on one of these rebases. Here is patch v017 to address your feedback. I also fixed the whitespace warning from the last Jenkins run.

          BTW, the whitespace check is triggering on any lines with trailing whitespace in the patch. This includes diff context on lines that the patch isn't really changing. Do you think we ought to change this to match against only lines that start with '+'?

          Show
          cnauroth Chris Nauroth added a comment - Thank you for the review, Allen. You're right about passing the parameter incorrectly to Maven. I think I introduced that bug on one of these rebases. Here is patch v017 to address your feedback. I also fixed the whitespace warning from the last Jenkins run. BTW, the whitespace check is triggering on any lines with trailing whitespace in the patch. This includes diff context on lines that the patch isn't really changing. Do you think we ought to change this to match against only lines that start with '+'?
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to test-patch or smart-apply-patch has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6881/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6881/console in case of problems.
          Hide
          aw Allen Wittenauer added a comment -

          BTW, the whitespace check is triggering on any lines with trailing whitespace in the patch. This includes diff context on lines that the patch isn't really changing. Do you think we ought to change this to match against only lines that start with '+'?

          That's not actually how it works... but there is a math bug I haven't had a chance to track down yet. I do know what it causes it (and checkstyle) to skip new files. Got that one fixed in one of my other test-patch patches.

          Show
          aw Allen Wittenauer added a comment - BTW, the whitespace check is triggering on any lines with trailing whitespace in the patch. This includes diff context on lines that the patch isn't really changing. Do you think we ought to change this to match against only lines that start with '+'? That's not actually how it works... but there is a math bug I haven't had a chance to track down yet. I do know what it causes it (and checkstyle) to skip new files. Got that one fixed in one of my other test-patch patches.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 19m 39s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 tests included 0m 0s The patch appears to include 12 new or modified test files.
          +1 javac 7m 31s There were no new javac warning messages.
          +1 javadoc 9m 41s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 54s There were no new checkstyle issues.
          +1 shellcheck 0m 9s There were no new shellcheck (v0.3.3) issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 38s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          +1 findbugs 5m 2s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 6m 50s Tests passed in hadoop-common.
          -1 hdfs tests 42m 16s Tests failed in hadoop-hdfs.
              96m 43s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestAppendSnapshotTruncate
            hadoop.hdfs.server.namenode.ha.TestHAAppend



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12736383/HADOOP-11984.017.patch
          Optional Tests shellcheck javadoc javac unit findbugs checkstyle
          git revision trunk / a8acdd6
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6881/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6881/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6881/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6881/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 19m 39s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 tests included 0m 0s The patch appears to include 12 new or modified test files. +1 javac 7m 31s There were no new javac warning messages. +1 javadoc 9m 41s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 54s There were no new checkstyle issues. +1 shellcheck 0m 9s There were no new shellcheck (v0.3.3) issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 38s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 5m 2s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 6m 50s Tests passed in hadoop-common. -1 hdfs tests 42m 16s Tests failed in hadoop-hdfs.     96m 43s   Reason Tests Failed unit tests hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.server.namenode.ha.TestHAAppend Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12736383/HADOOP-11984.017.patch Optional Tests shellcheck javadoc javac unit findbugs checkstyle git revision trunk / a8acdd6 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6881/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6881/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6881/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6881/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Something seems off about TestHAAppend. I increased the timeout from 30 seconds to 60 seconds, but it still failed. I think this one will need more investigation.

          Show
          cnauroth Chris Nauroth added a comment - Something seems off about TestHAAppend . I increased the timeout from 30 seconds to 60 seconds, but it still failed. I think this one will need more investigation.
          Hide
          aw Allen Wittenauer added a comment -

          We should remove MaxPerm since it isn't supported in Java 8 anymore.

          Show
          aw Allen Wittenauer added a comment - We should remove MaxPerm since it isn't supported in Java 8 anymore.
          Hide
          aw Allen Wittenauer added a comment -

          An experiment:

          -18:

          • same as -17, but modified enough to trigger the new test-patch on the yetus branch....

          .. because if you want to do some testing, go big or go home.

          Show
          aw Allen Wittenauer added a comment - An experiment: -18: same as -17, but modified enough to trigger the new test-patch on the yetus branch.... .. because if you want to do some testing, go big or go home.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to the files used for the QA process has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7121/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to the files used for the QA process has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7121/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s precommit patch detected.
          +1 mvninstall 7m 53s HADOOP-12111 passed
          +1 javac 1m 21s HADOOP-12111 passed with JDK v1.8.0
          +1 javac 1m 19s HADOOP-12111 passed with JDK v1.7.0_55
          +1 javadoc 2m 2s HADOOP-12111 passed with JDK v1.8.0
          +1 javadoc 2m 55s HADOOP-12111 passed with JDK v1.7.0_55
          0 @author 15m 32s Skipping @author checks as test-patch.sh has been patched.
          +1 test4tests 0m 0s The patch appears to include 12 new or modified test files.
          +1 checkstyle 1m 54s HADOOP-12111 passed
          -1 findbugs 3m 7s hadoop-hdfs-project/hadoop-hdfs in HADOOP-12111 cannot run convertXmlToText from findbugs
          +1 javac 1m 29s the patch passed with JDK v1.8.0
          +1 javac 1m 20s the patch passed with JDK v1.7.0_55
          +1 javadoc 2m 3s the patch passed with JDK v1.8.0
          +1 javadoc 2m 59s the patch passed with JDK v1.7.0_55
          +1 asflicense 0m 20s Patch does not generate ASF License warnings.
          -1 checkstyle 0m 37s Patch generated 8 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 0, now 8).
          +1 shellcheck 0m 9s There were no new shellcheck issues.
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          -1 mvninstall 0m 52s hadoop-hdfs in the patch failed.
          +1 eclipse 0m 23s the patch passed
          +1 findbugs 5m 10s the patch passed
          -1 unit 18m 49s hadoop-common in the patch failed with JDK v1.8.0.
          -1 unit 45m 51s hadoop-hdfs in the patch failed with JDK v1.8.0.
          -1 unit 18m 51s hadoop-common in the patch failed with JDK v1.7.0_55.
          -1 unit 42m 43s hadoop-hdfs in the patch failed with JDK v1.7.0_55.
          166m 42s



          Reason Tests
          JDK v1.8.0 Failed unit tests hadoop.ha.TestZKFailoverControllerStress
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.TestDistributedFileSystem
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          JDK v1.8.0 Timed out tests org.apache.hadoop.http.TestAuthenticationSessionCookie
          JDK v1.7.0_55 Failed unit tests hadoop.fs.shell.TestTextCommand
            hadoop.hdfs.TestReplaceDatanodeOnFailure
            hadoop.hdfs.TestEncryptionZonesWithKMS
          JDK v1.7.0_55 Timed out tests org.apache.hadoop.http.TestAuthenticationSessionCookie



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12742971/HADOOP-11984.HADOOP-12111.018.patch
          git revision HADOOP-12111 / b4c7aa1
          Optional Tests asflicense shellcheck javac javadoc mvninstall unit xml findbugs checkstyle
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/dev-support-test/personality/hadoop.sh
          Default Java 1.7.0_55
          Multi-JDK versions /home/jenkins/tools/java/jdk1.8.0:1.8.0 /home/jenkins/tools/java/jdk1.7.0_55:1.7.0_55
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/7121/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          shellcheck v0.3.3 (This is an old version that has serious bugs. Consider upgrading.)
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7121/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/7121/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/7121/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/7121/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/7121/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_55.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/7121/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_55.txt
          JDK v1.7.0_55 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7121/testReport/
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7121/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s precommit patch detected. +1 mvninstall 7m 53s HADOOP-12111 passed +1 javac 1m 21s HADOOP-12111 passed with JDK v1.8.0 +1 javac 1m 19s HADOOP-12111 passed with JDK v1.7.0_55 +1 javadoc 2m 2s HADOOP-12111 passed with JDK v1.8.0 +1 javadoc 2m 55s HADOOP-12111 passed with JDK v1.7.0_55 0 @author 15m 32s Skipping @author checks as test-patch.sh has been patched. +1 test4tests 0m 0s The patch appears to include 12 new or modified test files. +1 checkstyle 1m 54s HADOOP-12111 passed -1 findbugs 3m 7s hadoop-hdfs-project/hadoop-hdfs in HADOOP-12111 cannot run convertXmlToText from findbugs +1 javac 1m 29s the patch passed with JDK v1.8.0 +1 javac 1m 20s the patch passed with JDK v1.7.0_55 +1 javadoc 2m 3s the patch passed with JDK v1.8.0 +1 javadoc 2m 59s the patch passed with JDK v1.7.0_55 +1 asflicense 0m 20s Patch does not generate ASF License warnings. -1 checkstyle 0m 37s Patch generated 8 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 0, now 8). +1 shellcheck 0m 9s There were no new shellcheck issues. +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. -1 mvninstall 0m 52s hadoop-hdfs in the patch failed. +1 eclipse 0m 23s the patch passed +1 findbugs 5m 10s the patch passed -1 unit 18m 49s hadoop-common in the patch failed with JDK v1.8.0. -1 unit 45m 51s hadoop-hdfs in the patch failed with JDK v1.8.0. -1 unit 18m 51s hadoop-common in the patch failed with JDK v1.7.0_55. -1 unit 42m 43s hadoop-hdfs in the patch failed with JDK v1.7.0_55. 166m 42s Reason Tests JDK v1.8.0 Failed unit tests hadoop.ha.TestZKFailoverControllerStress   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots JDK v1.8.0 Timed out tests org.apache.hadoop.http.TestAuthenticationSessionCookie JDK v1.7.0_55 Failed unit tests hadoop.fs.shell.TestTextCommand   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.TestEncryptionZonesWithKMS JDK v1.7.0_55 Timed out tests org.apache.hadoop.http.TestAuthenticationSessionCookie Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12742971/HADOOP-11984.HADOOP-12111.018.patch git revision HADOOP-12111 / b4c7aa1 Optional Tests asflicense shellcheck javac javadoc mvninstall unit xml findbugs checkstyle uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/dev-support-test/personality/hadoop.sh Default Java 1.7.0_55 Multi-JDK versions /home/jenkins/tools/java/jdk1.8.0:1.8.0 /home/jenkins/tools/java/jdk1.7.0_55:1.7.0_55 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/7121/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html shellcheck v0.3.3 (This is an old version that has serious bugs. Consider upgrading.) checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7121/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/7121/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/7121/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/7121/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/7121/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_55.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/7121/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_55.txt JDK v1.7.0_55 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7121/testReport/ Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7121/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Just rebased the .017 patch.

          This is a very important work to be done towards build time.
          Thanks Chris Nauroth for the effort.

          Show
          vinayrpet Vinayakumar B added a comment - Just rebased the .017 patch. This is a very important work to be done towards build time. Thanks Chris Nauroth for the effort.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to the files used for the QA process has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7690/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to the files used for the QA process has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7690/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 19m 18s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 tests included 0m 0s The patch appears to include 12 new or modified test files.
          +1 javac 7m 51s There were no new javac warning messages.
          +1 javadoc 9m 54s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 5s There were no new checkstyle issues.
          +1 shellcheck 0m 9s There were no new shellcheck (v0.3.3) issues.
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 17s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 16m 45s Tests failed in hadoop-common.
          -1 hdfs tests 58m 39s Tests failed in hadoop-hdfs.
              121m 33s  



          Reason Tests
          Timed out tests org.apache.hadoop.http.TestAuthenticationSessionCookie
            org.apache.hadoop.hdfs.web.oauth2.TestClientCredentialTimeBasedTokenRefresher
            org.apache.hadoop.fs.TestSWebHdfsFileContextMainOperations



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12761888/HADOOP-11984.019.patch
          Optional Tests shellcheck javadoc javac unit findbugs checkstyle
          git revision trunk / a2c76e5
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7690/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7690/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7690/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7690/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 19m 18s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 tests included 0m 0s The patch appears to include 12 new or modified test files. +1 javac 7m 51s There were no new javac warning messages. +1 javadoc 9m 54s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 5s There were no new checkstyle issues. +1 shellcheck 0m 9s There were no new shellcheck (v0.3.3) issues. +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 17s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 16m 45s Tests failed in hadoop-common. -1 hdfs tests 58m 39s Tests failed in hadoop-hdfs.     121m 33s   Reason Tests Timed out tests org.apache.hadoop.http.TestAuthenticationSessionCookie   org.apache.hadoop.hdfs.web.oauth2.TestClientCredentialTimeBasedTokenRefresher   org.apache.hadoop.fs.TestSWebHdfsFileContextMainOperations Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12761888/HADOOP-11984.019.patch Optional Tests shellcheck javadoc javac unit findbugs checkstyle git revision trunk / a2c76e5 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7690/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7690/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7690/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7690/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Updated patch for failures.

          Used ServerSocketUtil to findout free ports instead of LOCKDIR

          Show
          vinayrpet Vinayakumar B added a comment - Updated patch for failures. Used ServerSocketUtil to findout free ports instead of LOCKDIR
          Hide
          wheat9 Haohui Mai added a comment -

          I think the patch looks pretty good overall. +1 pending Jenkins.

          Show
          wheat9 Haohui Mai added a comment - I think the patch looks pretty good overall. +1 pending Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to the files used for the QA process has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7693/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to the files used for the QA process has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7693/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 20m 16s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 tests included 0m 0s The patch appears to include 15 new or modified test files.
          +1 javac 8m 8s There were no new javac warning messages.
          +1 javadoc 10m 30s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 15s There were no new checkstyle issues.
          +1 shellcheck 0m 11s There were no new shellcheck (v0.3.3) issues.
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 4m 27s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 7m 22s Tests failed in hadoop-common.
          -1 hdfs tests 53m 31s Tests failed in hadoop-hdfs.
              109m 20s  



          Reason Tests
          Failed unit tests hadoop.ha.TestZKFailoverController
            hadoop.ha.TestActiveStandbyElector
            hadoop.ha.TestActiveStandbyElectorRealZK
            hadoop.ha.TestZKFailoverControllerStress
            hadoop.hdfs.tools.TestDFSZKFailoverController
            hadoop.hdfs.TestLeaseRecovery2
            hadoop.fs.TestSWebHdfsFileContextMainOperations
            hadoop.hdfs.server.blockmanagement.TestNodeCount



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12761964/HADOOP-11984.020.patch
          Optional Tests shellcheck javadoc javac unit findbugs checkstyle
          git revision trunk / f3e5bc6
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7693/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7693/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7693/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7693/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 20m 16s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 tests included 0m 0s The patch appears to include 15 new or modified test files. +1 javac 8m 8s There were no new javac warning messages. +1 javadoc 10m 30s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 15s There were no new checkstyle issues. +1 shellcheck 0m 11s There were no new shellcheck (v0.3.3) issues. +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 4m 27s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 7m 22s Tests failed in hadoop-common. -1 hdfs tests 53m 31s Tests failed in hadoop-hdfs.     109m 20s   Reason Tests Failed unit tests hadoop.ha.TestZKFailoverController   hadoop.ha.TestActiveStandbyElector   hadoop.ha.TestActiveStandbyElectorRealZK   hadoop.ha.TestZKFailoverControllerStress   hadoop.hdfs.tools.TestDFSZKFailoverController   hadoop.hdfs.TestLeaseRecovery2   hadoop.fs.TestSWebHdfsFileContextMainOperations   hadoop.hdfs.server.blockmanagement.TestNodeCount Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12761964/HADOOP-11984.020.patch Optional Tests shellcheck javadoc javac unit findbugs checkstyle git revision trunk / f3e5bc6 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7693/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7693/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7693/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7693/console This message was automatically generated.
          Hide
          wheat9 Haohui Mai added a comment -

          Vinayakumar B it looks like the failures are related to the latest patch. Can you please double check?

          Show
          wheat9 Haohui Mai added a comment - Vinayakumar B it looks like the failures are related to the latest patch. Can you please double check?
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to the files used for the QA process has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7698/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to the files used for the QA process has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7698/console in case of problems.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Updated the patch to fix failures.
          TestLeaseRecovery2 and TestNodeCount are not related. Others are passing in local with parallel-tests with updated patch.

          Show
          vinayrpet Vinayakumar B added a comment - Updated the patch to fix failures. TestLeaseRecovery2 and TestNodeCount are not related. Others are passing in local with parallel-tests with updated patch.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          -1 pre-patch 18m 6s Findbugs (version ) appears to be broken on trunk.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 tests included 0m 0s The patch appears to include 16 new or modified test files.
          +1 javac 7m 58s There were no new javac warning messages.
          +1 javadoc 10m 11s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 37s There were no new checkstyle issues.
          +1 shellcheck 0m 9s There were no new shellcheck (v0.3.3) issues.
          +1 whitespace 0m 3s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 4m 24s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 8m 12s Tests passed in hadoop-common.
          -1 hdfs tests 42m 3s Tests failed in hadoop-hdfs.
              95m 20s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.TestReplaceDatanodeOnFailure
          Timed out tests org.apache.hadoop.hdfs.TestBalancerBandwidth
            org.apache.hadoop.hdfs.TestDecommission
            org.apache.hadoop.hdfs.TestSetrepIncreasing
            org.apache.hadoop.hdfs.TestDFSUpgradeFromImage



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12762079/HADOOP-11984.021.patch
          Optional Tests shellcheck javadoc javac unit findbugs checkstyle
          git revision trunk / ead1b9e
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7698/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7698/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7698/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7698/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. -1 pre-patch 18m 6s Findbugs (version ) appears to be broken on trunk. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 tests included 0m 0s The patch appears to include 16 new or modified test files. +1 javac 7m 58s There were no new javac warning messages. +1 javadoc 10m 11s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 37s There were no new checkstyle issues. +1 shellcheck 0m 9s There were no new shellcheck (v0.3.3) issues. +1 whitespace 0m 3s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 4m 24s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 8m 12s Tests passed in hadoop-common. -1 hdfs tests 42m 3s Tests failed in hadoop-hdfs.     95m 20s   Reason Tests Failed unit tests hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.TestReplaceDatanodeOnFailure Timed out tests org.apache.hadoop.hdfs.TestBalancerBandwidth   org.apache.hadoop.hdfs.TestDecommission   org.apache.hadoop.hdfs.TestSetrepIncreasing   org.apache.hadoop.hdfs.TestDFSUpgradeFromImage Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762079/HADOOP-11984.021.patch Optional Tests shellcheck javadoc javac unit findbugs checkstyle git revision trunk / ead1b9e hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7698/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7698/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7698/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7698/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          All above tests passed locally with parallel tests profile enabled.

          But I am hitting some other different failures everytime.
          So I think this patch needs some more runs to make stable run.

          hadoop-common module seems stable with parallel tests. But hadoop-hdfs is little giving some failures.
          I am trying to run locally. Meanwhile going to trigger on more jenkins on latest patch.

          Show
          vinayrpet Vinayakumar B added a comment - All above tests passed locally with parallel tests profile enabled. But I am hitting some other different failures everytime. So I think this patch needs some more runs to make stable run. hadoop-common module seems stable with parallel tests. But hadoop-hdfs is little giving some failures. I am trying to run locally. Meanwhile going to trigger on more jenkins on latest patch.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to the files used for the QA process has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7701/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to the files used for the QA process has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7701/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 19m 35s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 tests included 0m 0s The patch appears to include 16 new or modified test files.
          +1 javac 7m 41s There were no new javac warning messages.
          +1 javadoc 10m 11s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 8s There were no new checkstyle issues.
          +1 shellcheck 0m 9s There were no new shellcheck (v0.3.3) issues.
          +1 whitespace 0m 3s The patch has no lines that end in whitespace.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 4m 23s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 8m 13s Tests passed in hadoop-common.
          -1 hdfs tests 45m 6s Tests failed in hadoop-hdfs.
              100m 3s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
            hadoop.hdfs.server.mover.TestStorageMover
            hadoop.hdfs.web.TestWebHdfsTokens
            hadoop.hdfs.web.TestWebHDFSOAuth2
            hadoop.fs.TestSWebHdfsFileContextMainOperations
            hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
            hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.server.namenode.TestFileLimit
            hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
            hadoop.hdfs.web.TestHttpsFileSystem
            hadoop.hdfs.server.namenode.TestINodeFile
          Timed out tests org.apache.hadoop.hdfs.web.oauth2.TestClientCredentialTimeBasedTokenRefresher



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12762079/HADOOP-11984.021.patch
          Optional Tests shellcheck javadoc javac unit findbugs checkstyle
          git revision trunk / e52bc69
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7701/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7701/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7701/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7701/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 19m 35s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 tests included 0m 0s The patch appears to include 16 new or modified test files. +1 javac 7m 41s There were no new javac warning messages. +1 javadoc 10m 11s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 8s There were no new checkstyle issues. +1 shellcheck 0m 9s There were no new shellcheck (v0.3.3) issues. +1 whitespace 0m 3s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 4m 23s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 8m 13s Tests passed in hadoop-common. -1 hdfs tests 45m 6s Tests failed in hadoop-hdfs.     100m 3s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.server.mover.TestStorageMover   hadoop.hdfs.web.TestWebHdfsTokens   hadoop.hdfs.web.TestWebHDFSOAuth2   hadoop.fs.TestSWebHdfsFileContextMainOperations   hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer   hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.TestFileLimit   hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer   hadoop.hdfs.web.TestHttpsFileSystem   hadoop.hdfs.server.namenode.TestINodeFile Timed out tests org.apache.hadoop.hdfs.web.oauth2.TestClientCredentialTimeBasedTokenRefresher Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762079/HADOOP-11984.021.patch Optional Tests shellcheck javadoc javac unit findbugs checkstyle git revision trunk / e52bc69 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7701/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7701/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7701/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7701/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to the files used for the QA process has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7703/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to the files used for the QA process has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7703/console in case of problems.
          Hide
          cnauroth Chris Nauroth added a comment -

          Vinayakumar B, thank you very much for your help with rebasing this and trying to complete it.

          But I am hitting some other different failures everytime.

          Yes, this is exactly what I had been seeing. We're currently on revision 21 of this patch, and each one has gotten a bit closer, but I still don't get consistently clean runs. Progress is slow, because you effectively need to do full test runs in parallel mode to try to get a local repro and then fix root cause.

          Of course, we have existing problems with stability of our test runs even in single-process mode. It's hard to tell if parallel execution is really introducing the problems.

          hadoop-common module seems stable with parallel tests. But hadoop-hdfs is little giving some failures.

          Maybe a good plan would be:

          1. Split off a patch covering Hadoop Common only. We're probably close to stability on that part, and there is still a modest performance improvement from doing that.
          2. Split off some of the individual test fixes into their own small separate patches and get them committed. As was discussed on the dev list, these fixes have merit on their own, even without enabling parallel tests.
          3. Track a separate HDFS JIRA for parallel execution of HDFS tests, and keep working there.

          Please let me know your thoughts on the plan. If it sounds good, I'd be happy to take care of the JIRA red tape and split up the patches.

          Show
          cnauroth Chris Nauroth added a comment - Vinayakumar B , thank you very much for your help with rebasing this and trying to complete it. But I am hitting some other different failures everytime. Yes, this is exactly what I had been seeing. We're currently on revision 21 of this patch, and each one has gotten a bit closer, but I still don't get consistently clean runs. Progress is slow, because you effectively need to do full test runs in parallel mode to try to get a local repro and then fix root cause. Of course, we have existing problems with stability of our test runs even in single-process mode. It's hard to tell if parallel execution is really introducing the problems. hadoop-common module seems stable with parallel tests. But hadoop-hdfs is little giving some failures. Maybe a good plan would be: Split off a patch covering Hadoop Common only. We're probably close to stability on that part, and there is still a modest performance improvement from doing that. Split off some of the individual test fixes into their own small separate patches and get them committed. As was discussed on the dev list, these fixes have merit on their own, even without enabling parallel tests. Track a separate HDFS JIRA for parallel execution of HDFS tests, and keep working there. Please let me know your thoughts on the plan. If it sounds good, I'd be happy to take care of the JIRA red tape and split up the patches.
          Hide
          wheat9 Haohui Mai added a comment -

          Thanks Chris Nauroth. I think it makes a lot of sense.

          I'm leaning towards to put it in HDFS as well. The sets of failed tests are fairly consistent with what I have seen in the HDFS-6200 work. Having this patch in obviously does not fix the tests, but it'll reduce the turn around time dramatically (saving 3hrs!) which makes fixing these tests more productive.

          Show
          wheat9 Haohui Mai added a comment - Thanks Chris Nauroth . I think it makes a lot of sense. I'm leaning towards to put it in HDFS as well. The sets of failed tests are fairly consistent with what I have seen in the HDFS-6200 work. Having this patch in obviously does not fix the tests, but it'll reduce the turn around time dramatically (saving 3hrs!) which makes fixing these tests more productive.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Please let me know your thoughts on the plan. If it sounds good, I'd be happy to take care of the JIRA red tape and split up the patches.

          Thanks Chris Nauroth, this sounds reasonable.
          I would be happy to help in stabilizing the HDFS part of parallel test execution.

          Show
          vinayrpet Vinayakumar B added a comment - Please let me know your thoughts on the plan. If it sounds good, I'd be happy to take care of the JIRA red tape and split up the patches. Thanks Chris Nauroth , this sounds reasonable. I would be happy to help in stabilizing the HDFS part of parallel test execution.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Having this patch in obviously does not fix the tests, but it'll reduce the turn around time dramatically (saving 3hrs!) which makes fixing these tests more productive

          Correct. Its obviously reduces turn-around time.
          But I feel everytime, there might be a new set of failures. So everytime someone has to analyze to check whether they are by the patch or not. There could be a chance of ignoring those failures.

          Show
          vinayrpet Vinayakumar B added a comment - Having this patch in obviously does not fix the tests, but it'll reduce the turn around time dramatically (saving 3hrs!) which makes fixing these tests more productive Correct. Its obviously reduces turn-around time. But I feel everytime, there might be a new set of failures. So everytime someone has to analyze to check whether they are by the patch or not. There could be a chance of ignoring those failures.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 19m 39s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 tests included 0m 0s The patch appears to include 16 new or modified test files.
          +1 javac 7m 49s There were no new javac warning messages.
          +1 javadoc 10m 4s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 10s There were no new checkstyle issues.
          +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues.
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 7m 52s Tests failed in hadoop-common.
          +1 hdfs tests 42m 34s Tests passed in hadoop-hdfs.
              97m 21s  



          Reason Tests
          Failed unit tests hadoop.metrics2.impl.TestGangliaMetrics
            hadoop.ipc.TestRPC



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12762079/HADOOP-11984.021.patch
          Optional Tests shellcheck javadoc javac unit findbugs checkstyle
          git revision trunk / 71a81b6
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7703/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7703/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7703/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7703/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 19m 39s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 tests included 0m 0s The patch appears to include 16 new or modified test files. +1 javac 7m 49s There were no new javac warning messages. +1 javadoc 10m 4s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 10s There were no new checkstyle issues. +1 shellcheck 0m 8s There were no new shellcheck (v0.3.3) issues. +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 7m 52s Tests failed in hadoop-common. +1 hdfs tests 42m 34s Tests passed in hadoop-hdfs.     97m 21s   Reason Tests Failed unit tests hadoop.metrics2.impl.TestGangliaMetrics   hadoop.ipc.TestRPC Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762079/HADOOP-11984.021.patch Optional Tests shellcheck javadoc javac unit findbugs checkstyle git revision trunk / 71a81b6 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7703/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7703/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7703/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7703/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          This is excellent Run.

          This is also possible if we are lucky enough :-P

          Show
          vinayrpet Vinayakumar B added a comment - This is excellent Run. This is also possible if we are lucky enough :-P
          Hide
          vinayrpet Vinayakumar B added a comment -

          I have separated the hadoop-common part of the patch.
          Also changed the 'parallel-tests' profile name to 'parallel-tests-disabled' in hadoop-hdfs/pom.xml to not to run HDFS tests in parallel now.
          That will be enabled again in follow up jira for HDFS tests' parallel run.

          Haohui Mai and Chris Nauroth, is this looks fine to you?

          Show
          vinayrpet Vinayakumar B added a comment - I have separated the hadoop-common part of the patch. Also changed the 'parallel-tests' profile name to 'parallel-tests-disabled' in hadoop-hdfs/pom.xml to not to run HDFS tests in parallel now. That will be enabled again in follow up jira for HDFS tests' parallel run. Haohui Mai and Chris Nauroth , is this looks fine to you?
          Hide
          vinayrpet Vinayakumar B added a comment -

          Filed HDFS-9139 for HDFS tests' parallel run possible.

          Show
          vinayrpet Vinayakumar B added a comment - Filed HDFS-9139 for HDFS tests' parallel run possible.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to the files used for the QA process has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7704/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to the files used for the QA process has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7704/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          -1 pre-patch 17m 52s Findbugs (version ) appears to be broken on trunk.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 tests included 0m 0s The patch appears to include 7 new or modified test files.
          +1 javac 8m 3s There were no new javac warning messages.
          +1 javadoc 10m 18s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 39s There were no new checkstyle issues.
          +1 shellcheck 0m 9s There were no new shellcheck (v0.3.3) issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 37s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 27s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 6m 36s Tests failed in hadoop-common.
          -1 hdfs tests 161m 32s Tests failed in hadoop-hdfs.
              213m 15s  



          Reason Tests
          Failed unit tests hadoop.security.ssl.TestReloadingX509TrustManager
            hadoop.fs.shell.TestTextCommand
            hadoop.metrics2.impl.TestGangliaMetrics
            hadoop.net.TestDNS
            hadoop.hdfs.TestParallelRead
            hadoop.hdfs.web.TestWebHDFSOAuth2
          Timed out tests org.apache.hadoop.hdfs.TestDecommission
            org.apache.hadoop.cli.TestHDFSCLI



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12762212/HADOOP-11984.022.patch
          Optional Tests shellcheck javadoc javac unit findbugs checkstyle
          git revision trunk / d1b9b85
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7704/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7704/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7704/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7704/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. -1 pre-patch 17m 52s Findbugs (version ) appears to be broken on trunk. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 tests included 0m 0s The patch appears to include 7 new or modified test files. +1 javac 8m 3s There were no new javac warning messages. +1 javadoc 10m 18s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 39s There were no new checkstyle issues. +1 shellcheck 0m 9s There were no new shellcheck (v0.3.3) issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 27s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 6m 36s Tests failed in hadoop-common. -1 hdfs tests 161m 32s Tests failed in hadoop-hdfs.     213m 15s   Reason Tests Failed unit tests hadoop.security.ssl.TestReloadingX509TrustManager   hadoop.fs.shell.TestTextCommand   hadoop.metrics2.impl.TestGangliaMetrics   hadoop.net.TestDNS   hadoop.hdfs.TestParallelRead   hadoop.hdfs.web.TestWebHDFSOAuth2 Timed out tests org.apache.hadoop.hdfs.TestDecommission   org.apache.hadoop.cli.TestHDFSCLI Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762212/HADOOP-11984.022.patch Optional Tests shellcheck javadoc javac unit findbugs checkstyle git revision trunk / d1b9b85 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7704/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7704/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7704/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7704/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          removed change in hdfs pom file

          Show
          vinayrpet Vinayakumar B added a comment - removed change in hdfs pom file
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to the files used for the QA process has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7706/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to the files used for the QA process has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7706/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 17m 37s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 tests included 0m 0s The patch appears to include 7 new or modified test files.
          +1 javac 8m 3s There were no new javac warning messages.
          +1 javadoc 10m 16s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 12s There were no new checkstyle issues.
          +1 shellcheck 0m 9s There were no new shellcheck (v0.3.3) issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 1m 51s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 6m 19s Tests failed in hadoop-common.
              47m 58s  



          Reason Tests
          Failed unit tests hadoop.metrics2.sink.TestFileSink



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12762446/HADOOP-11984.023.patch
          Optional Tests shellcheck javadoc javac unit findbugs checkstyle
          git revision trunk / 83e99d0
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7706/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7706/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7706/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 17m 37s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 tests included 0m 0s The patch appears to include 7 new or modified test files. +1 javac 8m 3s There were no new javac warning messages. +1 javadoc 10m 16s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 12s There were no new checkstyle issues. +1 shellcheck 0m 9s There were no new shellcheck (v0.3.3) issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 51s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 6m 19s Tests failed in hadoop-common.     47m 58s   Reason Tests Failed unit tests hadoop.metrics2.sink.TestFileSink Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762446/HADOOP-11984.023.patch Optional Tests shellcheck javadoc javac unit findbugs checkstyle git revision trunk / 83e99d0 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7706/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7706/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7706/console This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Can I highlight that the common junit test run phase took 6m36, whereas one of mine last week took 22m 44s;

          Show
          stevel@apache.org Steve Loughran added a comment - Can I highlight that the common junit test run phase took 6m36, whereas one of mine last week took 22m 44s;
          Hide
          vinayrpet Vinayakumar B added a comment -

          Steve Loughran, Yes, thats excellent improvement in runtime.
          I just hope test results also stays stable for all runs.

          Show
          vinayrpet Vinayakumar B added a comment - Steve Loughran , Yes, thats excellent improvement in runtime. I just hope test results also stays stable for all runs.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to the files used for the QA process has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7711/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to the files used for the QA process has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7711/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 16m 52s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 tests included 0m 0s The patch appears to include 7 new or modified test files.
          +1 javac 7m 47s There were no new javac warning messages.
          +1 javadoc 9m 56s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 4s There were no new checkstyle issues.
          +1 shellcheck 0m 10s There were no new shellcheck (v0.3.3) issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 30s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 7m 41s Tests failed in hadoop-common.
              47m 51s  



          Reason Tests
          Failed unit tests hadoop.crypto.key.TestValueQueue



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12762446/HADOOP-11984.023.patch
          Optional Tests shellcheck javadoc javac unit findbugs checkstyle
          git revision trunk / 861b52d
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7711/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7711/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7711/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 16m 52s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 tests included 0m 0s The patch appears to include 7 new or modified test files. +1 javac 7m 47s There were no new javac warning messages. +1 javadoc 9m 56s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 4s There were no new checkstyle issues. +1 shellcheck 0m 10s There were no new shellcheck (v0.3.3) issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 30s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 7m 41s Tests failed in hadoop-common.     47m 51s   Reason Tests Failed unit tests hadoop.crypto.key.TestValueQueue Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762446/HADOOP-11984.023.patch Optional Tests shellcheck javadoc javac unit findbugs checkstyle git revision trunk / 861b52d hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7711/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7711/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7711/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Chris Nauroth and Haohui Mai, do you think we can commit this now ?
          As it has shown somewhat stability

          Show
          vinayrpet Vinayakumar B added a comment - Chris Nauroth and Haohui Mai , do you think we can commit this now ? As it has shown somewhat stability
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to the files used for the QA process has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7712/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to the files used for the QA process has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7712/console in case of problems.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 17m 13s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 tests included 0m 0s The patch appears to include 7 new or modified test files.
          +1 javac 7m 55s There were no new javac warning messages.
          +1 javadoc 10m 9s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 8s There were no new checkstyle issues.
          +1 shellcheck 0m 9s There were no new shellcheck (v0.3.3) issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 1m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 7m 32s Tests failed in hadoop-common.
              48m 27s  



          Reason Tests
          Failed unit tests hadoop.fs.shell.TestCopyPreserveFlag



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12762446/HADOOP-11984.023.patch
          Optional Tests shellcheck javadoc javac unit findbugs checkstyle
          git revision trunk / bf37d3d
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7712/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7712/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7712/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 17m 13s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 tests included 0m 0s The patch appears to include 7 new or modified test files. +1 javac 7m 55s There were no new javac warning messages. +1 javadoc 10m 9s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 8s There were no new checkstyle issues. +1 shellcheck 0m 9s There were no new shellcheck (v0.3.3) issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 7m 32s Tests failed in hadoop-common.     48m 27s   Reason Tests Failed unit tests hadoop.fs.shell.TestCopyPreserveFlag Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762446/HADOOP-11984.023.patch Optional Tests shellcheck javadoc javac unit findbugs checkstyle git revision trunk / bf37d3d hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7712/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7712/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7712/console This message was automatically generated.
          Hide
          wheat9 Haohui Mai added a comment -

          +1 the latest patch.

          Show
          wheat9 Haohui Mai added a comment - +1 the latest patch.
          Hide
          cnauroth Chris Nauroth added a comment -

          I agree that we're really close, and the ~4x reduction in time for hadoop-common tests highlighted by Steve sure is pleasant.

          I'm attaching patch v024 with 2 changes:

          1. pom.xml no longer defines test.build.shared.data. Vinay solved this problem differently by using ServerSocketUtil#getPort, so there is no longer a need for a shared directory for multiple JUnit processes to coordinate with lock files corresponding to ports.
          2. Since the HDFS fixes have been spun off into the separate HDFS-9139 JIRA, we need to be careful not to trigger -Pparallel-tests for HDFS builds until after that gets completed. To help move things forward for hadoop-common, I've introduced a temporary hack in test-patch.sh so that it only uses -Pparallel-tests for the hadoop-common build.
          Show
          cnauroth Chris Nauroth added a comment - I agree that we're really close, and the ~4x reduction in time for hadoop-common tests highlighted by Steve sure is pleasant. I'm attaching patch v024 with 2 changes: pom.xml no longer defines test.build.shared.data . Vinay solved this problem differently by using ServerSocketUtil#getPort , so there is no longer a need for a shared directory for multiple JUnit processes to coordinate with lock files corresponding to ports. Since the HDFS fixes have been spun off into the separate HDFS-9139 JIRA, we need to be careful not to trigger -Pparallel-tests for HDFS builds until after that gets completed. To help move things forward for hadoop-common, I've introduced a temporary hack in test-patch.sh so that it only uses -Pparallel-tests for the hadoop-common build.
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to the files used for the QA process has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7714/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to the files used for the QA process has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/7714/console in case of problems.
          Hide
          aw Allen Wittenauer added a comment -

          Be aware that since test-patch is being modified, it will try to run parallel tests for EVERY branch....

          Show
          aw Allen Wittenauer added a comment - Be aware that since test-patch is being modified, it will try to run parallel tests for EVERY branch....
          Hide
          vinayrpet Vinayakumar B added a comment -

          Thanks Chris Nauroth for the updated patch.

          +1, I think latest changes also looks good.

          I almost committed the last v23 patch, now will wait for another run

          Show
          vinayrpet Vinayakumar B added a comment - Thanks Chris Nauroth for the updated patch. +1, I think latest changes also looks good. I almost committed the last v23 patch, now will wait for another run
          Hide
          vinayrpet Vinayakumar B added a comment -

          Be aware that since test-patch is being modified, it will try to run parallel tests for EVERY branch....

          Oh Yes, thanks Allen Wittenauer.
          Didn't notice that.
          IMO, since these fixes going to both trunk/branch-2, this should be okay.
          And other ongoing feature branches also expected to be rabased timely, So this should be okay for them too.

          Show
          vinayrpet Vinayakumar B added a comment - Be aware that since test-patch is being modified, it will try to run parallel tests for EVERY branch.... Oh Yes, thanks Allen Wittenauer . Didn't notice that. IMO, since these fixes going to both trunk/branch-2, this should be okay. And other ongoing feature branches also expected to be rabased timely, So this should be okay for them too.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 17m 17s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          +1 tests included 0m 0s The patch appears to include 7 new or modified test files.
          +1 javac 7m 49s There were no new javac warning messages.
          +1 javadoc 9m 59s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 9s There were no new checkstyle issues.
          +1 shellcheck 0m 9s There were no new shellcheck (v0.3.3) issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 1m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 7m 28s Tests failed in hadoop-common.
              48m 12s  



          Reason Tests
          Failed unit tests hadoop.metrics2.impl.TestGangliaMetrics
            hadoop.fs.shell.TestCopyPreserveFlag



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12762580/HADOOP-11984.024.patch
          Optional Tests shellcheck javadoc javac unit findbugs checkstyle
          git revision trunk / 0b31c23
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7714/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7714/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7714/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 17m 17s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 tests included 0m 0s The patch appears to include 7 new or modified test files. +1 javac 7m 49s There were no new javac warning messages. +1 javadoc 9m 59s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 9s There were no new checkstyle issues. +1 shellcheck 0m 9s There were no new shellcheck (v0.3.3) issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 7m 28s Tests failed in hadoop-common.     48m 12s   Reason Tests Failed unit tests hadoop.metrics2.impl.TestGangliaMetrics   hadoop.fs.shell.TestCopyPreserveFlag Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762580/HADOOP-11984.024.patch Optional Tests shellcheck javadoc javac unit findbugs checkstyle git revision trunk / 0b31c23 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7714/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7714/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7714/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          +1 for the updated v24 patch.

          Show
          vinayrpet Vinayakumar B added a comment - +1 for the updated v24 patch.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Committed to trunk and branch-2.

          Thank you all.

          Show
          vinayrpet Vinayakumar B added a comment - Committed to trunk and branch-2. Thank you all.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8527 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8527/)
          HADOOP-11984. Enable parallel JUnit tests in pre-commit (Contributed by Chris Nauroth) (vinayakumarb: rev f0f984e4e63d0dbafe93062a122ee051330db301)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestAuthenticationSessionCookie.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/ServerSocketUtil.java
          • hadoop-common-project/hadoop-common/pom.xml
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpCookieFlag.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          • hadoop-project/pom.xml
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestSSLHttpServer.java
          • dev-support/test-patch.sh
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8527 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8527/ ) HADOOP-11984 . Enable parallel JUnit tests in pre-commit (Contributed by Chris Nauroth) (vinayakumarb: rev f0f984e4e63d0dbafe93062a122ee051330db301) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestAuthenticationSessionCookie.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/ServerSocketUtil.java hadoop-common-project/hadoop-common/pom.xml hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpCookieFlag.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java hadoop-project/pom.xml hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestSSLHttpServer.java dev-support/test-patch.sh hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #448 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/448/)
          HADOOP-11984. Enable parallel JUnit tests in pre-commit (Contributed by Chris Nauroth) (vinayakumarb: rev f0f984e4e63d0dbafe93062a122ee051330db301)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpCookieFlag.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/ServerSocketUtil.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestSSLHttpServer.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java
          • hadoop-common-project/hadoop-common/pom.xml
          • hadoop-project/pom.xml
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestAuthenticationSessionCookie.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • dev-support/test-patch.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #448 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/448/ ) HADOOP-11984 . Enable parallel JUnit tests in pre-commit (Contributed by Chris Nauroth) (vinayakumarb: rev f0f984e4e63d0dbafe93062a122ee051330db301) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpCookieFlag.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/ServerSocketUtil.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestSSLHttpServer.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java hadoop-common-project/hadoop-common/pom.xml hadoop-project/pom.xml hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestAuthenticationSessionCookie.java hadoop-common-project/hadoop-common/CHANGES.txt dev-support/test-patch.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1187 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1187/)
          HADOOP-11984. Enable parallel JUnit tests in pre-commit (Contributed by Chris Nauroth) (vinayakumarb: rev f0f984e4e63d0dbafe93062a122ee051330db301)

          • hadoop-common-project/hadoop-common/pom.xml
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestSSLHttpServer.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java
          • hadoop-project/pom.xml
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestAuthenticationSessionCookie.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • dev-support/test-patch.sh
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpCookieFlag.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/ServerSocketUtil.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1187 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1187/ ) HADOOP-11984 . Enable parallel JUnit tests in pre-commit (Contributed by Chris Nauroth) (vinayakumarb: rev f0f984e4e63d0dbafe93062a122ee051330db301) hadoop-common-project/hadoop-common/pom.xml hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestSSLHttpServer.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java hadoop-project/pom.xml hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestAuthenticationSessionCookie.java hadoop-common-project/hadoop-common/CHANGES.txt dev-support/test-patch.sh hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpCookieFlag.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/ServerSocketUtil.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #425 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/425/)
          HADOOP-11984. Enable parallel JUnit tests in pre-commit (Contributed by Chris Nauroth) (vinayakumarb: rev f0f984e4e63d0dbafe93062a122ee051330db301)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestAuthenticationSessionCookie.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java
          • hadoop-project/pom.xml
          • hadoop-common-project/hadoop-common/pom.xml
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpCookieFlag.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/ServerSocketUtil.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestSSLHttpServer.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          • dev-support/test-patch.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #425 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/425/ ) HADOOP-11984 . Enable parallel JUnit tests in pre-commit (Contributed by Chris Nauroth) (vinayakumarb: rev f0f984e4e63d0dbafe93062a122ee051330db301) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestAuthenticationSessionCookie.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java hadoop-project/pom.xml hadoop-common-project/hadoop-common/pom.xml hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpCookieFlag.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/ServerSocketUtil.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestSSLHttpServer.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java dev-support/test-patch.sh hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #454 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/454/)
          HADOOP-11984. Enable parallel JUnit tests in pre-commit (Contributed by Chris Nauroth) (vinayakumarb: rev f0f984e4e63d0dbafe93062a122ee051330db301)

          • hadoop-project/pom.xml
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/ServerSocketUtil.java
          • dev-support/test-patch.sh
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          • hadoop-common-project/hadoop-common/pom.xml
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestSSLHttpServer.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpCookieFlag.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestAuthenticationSessionCookie.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #454 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/454/ ) HADOOP-11984 . Enable parallel JUnit tests in pre-commit (Contributed by Chris Nauroth) (vinayakumarb: rev f0f984e4e63d0dbafe93062a122ee051330db301) hadoop-project/pom.xml hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/ServerSocketUtil.java dev-support/test-patch.sh hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java hadoop-common-project/hadoop-common/pom.xml hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestSSLHttpServer.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpCookieFlag.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestAuthenticationSessionCookie.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2365 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2365/)
          HADOOP-11984. Enable parallel JUnit tests in pre-commit (Contributed by Chris Nauroth) (vinayakumarb: rev f0f984e4e63d0dbafe93062a122ee051330db301)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestSSLHttpServer.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpCookieFlag.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestAuthenticationSessionCookie.java
          • hadoop-project/pom.xml
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java
          • dev-support/test-patch.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java
          • hadoop-common-project/hadoop-common/pom.xml
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/ServerSocketUtil.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2365 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2365/ ) HADOOP-11984 . Enable parallel JUnit tests in pre-commit (Contributed by Chris Nauroth) (vinayakumarb: rev f0f984e4e63d0dbafe93062a122ee051330db301) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestSSLHttpServer.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpCookieFlag.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestAuthenticationSessionCookie.java hadoop-project/pom.xml hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java dev-support/test-patch.sh hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java hadoop-common-project/hadoop-common/pom.xml hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/ServerSocketUtil.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2392 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2392/)
          HADOOP-11984. Enable parallel JUnit tests in pre-commit (Contributed by Chris Nauroth) (vinayakumarb: rev f0f984e4e63d0dbafe93062a122ee051330db301)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          • hadoop-common-project/hadoop-common/pom.xml
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpCookieFlag.java
          • dev-support/test-patch.sh
          • hadoop-project/pom.xml
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestSSLHttpServer.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/ServerSocketUtil.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestAuthenticationSessionCookie.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2392 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2392/ ) HADOOP-11984 . Enable parallel JUnit tests in pre-commit (Contributed by Chris Nauroth) (vinayakumarb: rev f0f984e4e63d0dbafe93062a122ee051330db301) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java hadoop-common-project/hadoop-common/pom.xml hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpCookieFlag.java dev-support/test-patch.sh hadoop-project/pom.xml hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestSSLHttpServer.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/ServerSocketUtil.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestAuthenticationSessionCookie.java
          Hide
          cnauroth Chris Nauroth added a comment -

          Thank you for the reviews, everyone. Vinay, thank you for committing.

          Unfortunately, it looks like TestCopyPreserveFlag had problems in the last few runs. Let's keep an eye on that one.

          It will be interesting to see if feature branches have been coding their test suites to be resilient to parallel execution. I suppose this is ultimately better though, because problems would be caught earlier on the feature branch before a merge back to trunk.

          Show
          cnauroth Chris Nauroth added a comment - Thank you for the reviews, everyone. Vinay, thank you for committing. Unfortunately, it looks like TestCopyPreserveFlag had problems in the last few runs. Let's keep an eye on that one. It will be interesting to see if feature branches have been coding their test suites to be resilient to parallel execution. I suppose this is ultimately better though, because problems would be caught earlier on the feature branch before a merge back to trunk.

            People

            • Assignee:
              cnauroth Chris Nauroth
              Reporter:
              cnauroth Chris Nauroth
            • Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development