Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      IBRs from thousands of DNs under load will degrade NN performance due to excessive write-lock contention from multiple IPC handler threads. The IBR processing is quick, so the lock contention may be reduced by coalescing multiple IBRs into a single write-lock transaction. The handlers will also be freed up faster for other operations.

      1. HDFS-9198-branch-2.7.patch
        41 kB
        Vinayakumar B
      2. HDFS-9198-Branch-2.8-withamend.diff
        40 kB
        Uma Maheswara Rao G
      3. HDFS-9198-branch2.patch
        32 kB
        Daryn Sharp
      4. HDFS-9198-Branch-2-withamend.diff
        40 kB
        Uma Maheswara Rao G
      5. HDFS-9198-trunk.patch
        38 kB
        Daryn Sharp
      6. HDFS-9198-trunk.patch
        38 kB
        Daryn Sharp
      7. HDFS-9198-trunk.patch
        36 kB
        Daryn Sharp
      8. HDFS-9198-trunk.patch
        36 kB
        Daryn Sharp
      9. HDFS-9198-trunk.patch
        31 kB
        Daryn Sharp

        Issue Links

          Activity

          Hide
          daryn Daryn Sharp added a comment -

          Incremental block reports are dumped into a queue for asynchronous processing by a background thread. This thread acquires the write lock and processes IBRs until the queue drains or a max lock hold is met. The max hold is 4ms which may seem high, but if the NN is backlogged that much, it's better to take the hit to catch up to avoid client issues.

          Full BR processing also uses the queuing in a synchronous manner. This helps preserve the ordering between the IBRs and full BRs from a node. Another reason for synchronous full BR processing is it may issue a finalize command.

          IBRs do not send commands so they can be async. However, in the unlikely event that an IBR fails, the DN currently re-queues the IBR, but now the DN always sees success. In practice an IBR fails if the DN is dead or unregistered. On the off-chance that an IBR fails for another reason, I added minimal support to force the DN to re-register which elicits a full BR for re-syncing.

          (The patches are trivially minor line number conflicts and a @VisibleForTesting)

          Show
          daryn Daryn Sharp added a comment - Incremental block reports are dumped into a queue for asynchronous processing by a background thread. This thread acquires the write lock and processes IBRs until the queue drains or a max lock hold is met. The max hold is 4ms which may seem high, but if the NN is backlogged that much, it's better to take the hit to catch up to avoid client issues. Full BR processing also uses the queuing in a synchronous manner. This helps preserve the ordering between the IBRs and full BRs from a node. Another reason for synchronous full BR processing is it may issue a finalize command. IBRs do not send commands so they can be async. However, in the unlikely event that an IBR fails, the DN currently re-queues the IBR, but now the DN always sees success. In practice an IBR fails if the DN is dead or unregistered. On the off-chance that an IBR fails for another reason, I added minimal support to force the DN to re-register which elicits a full BR for re-syncing. (The patches are trivially minor line number conflicts and a @VisibleForTesting)
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 6s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 8m 4s There were no new javac warning messages.
          +1 javadoc 10m 22s There were no new javadoc warning messages.
          -1 release audit 0m 15s The applied patch generated 1 release audit warnings.
          -1 checkstyle 1m 25s The applied patch generated 6 new checkstyle issues (total was 424, now 425).
          +1 whitespace 0m 6s The patch has no lines that end in whitespace.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          -1 findbugs 2m 33s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 16s Pre-build of native portion
          -1 hdfs tests 200m 1s Tests failed in hadoop-hdfs.
              246m 16s  



          Reason Tests
          FindBugs module:hadoop-hdfs
          Failed unit tests hadoop.hdfs.web.TestWebHdfsTimeouts
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestParallelRead
            hadoop.hdfs.server.namenode.TestDeadDatanode
            hadoop.hdfs.web.TestWebHdfsContentLength
            hadoop.hdfs.server.blockmanagement.TestPendingReplication
            hadoop.hdfs.server.datanode.TestIncrementalBrVariations



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12765032/HDFS-9198-trunk.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / b925cf1
          Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12794/artifact/patchprocess/patchReleaseAuditProblems.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12794/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12794/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12794/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12794/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12794/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 6s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 8m 4s There were no new javac warning messages. +1 javadoc 10m 22s There were no new javadoc warning messages. -1 release audit 0m 15s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 25s The applied patch generated 6 new checkstyle issues (total was 424, now 425). +1 whitespace 0m 6s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 2m 33s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 16s Pre-build of native portion -1 hdfs tests 200m 1s Tests failed in hadoop-hdfs.     246m 16s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestParallelRead   hadoop.hdfs.server.namenode.TestDeadDatanode   hadoop.hdfs.web.TestWebHdfsContentLength   hadoop.hdfs.server.blockmanagement.TestPendingReplication   hadoop.hdfs.server.datanode.TestIncrementalBrVariations Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12765032/HDFS-9198-trunk.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / b925cf1 Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12794/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12794/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12794/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12794/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12794/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12794/console This message was automatically generated.
          Hide
          daryn Daryn Sharp added a comment -

          Most of the test failures are a race condition from IBRs not be sync now. Will update shortly. All you watchers, any comments on the approach?

          Show
          daryn Daryn Sharp added a comment - Most of the test failures are a race condition from IBRs not be sync now. Will update shortly. All you watchers, any comments on the approach?
          Hide
          daryn Daryn Sharp added a comment -

          Took care of minor findbugs warning, cleanup up most of the silly style stuff. Some of the complaints about metrics I don't think are valid due to the annotation magic that occurs.

          Updated the tests to flush the block ops queue to prevent races.

          Changed the queue offer/add to offer/put.

          Show
          daryn Daryn Sharp added a comment - Took care of minor findbugs warning, cleanup up most of the silly style stuff. Some of the complaints about metrics I don't think are valid due to the annotation magic that occurs. Updated the tests to flush the block ops queue to prevent races. Changed the queue offer/add to offer/put.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12766881/HDFS-9198-trunk.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 8d2d3eb
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13013/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766881/HDFS-9198-trunk.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 8d2d3eb Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13013/console This message was automatically generated.
          Hide
          daryn Daryn Sharp added a comment -

          had to rebase

          Show
          daryn Daryn Sharp added a comment - had to rebase
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 20m 31s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 6 new or modified test files.
          +1 javac 9m 31s There were no new javac warning messages.
          +1 javadoc 11m 25s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 33s The applied patch generated 6 new checkstyle issues (total was 414, now 415).
          +1 whitespace 0m 8s The patch has no lines that end in whitespace.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 41s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 34s Pre-build of native portion
          -1 hdfs tests 66m 27s Tests failed in hadoop-hdfs.
              118m 30s  



          Reason Tests
          Failed unit tests hadoop.fs.loadGenerator.TestLoadGenerator
            hadoop.fs.TestFcHdfsCreateMkdir
            hadoop.hdfs.server.namenode.ha.TestDNFencing
            hadoop.hdfs.TestRecoverStripedFile
            hadoop.fs.TestSWebHdfsFileContextMainOperations
            hadoop.fs.TestUrlStreamHandler
            hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot
            hadoop.fs.viewfs.TestViewFsAtHdfsRoot
            hadoop.fs.permission.TestStickyBit
            hadoop.hdfs.web.TestWebHDFS
          Timed out tests org.apache.hadoop.fs.TestEnhancedByteBufferAccess
            org.apache.hadoop.fs.TestUrlStreamHandlerFactory
            org.apache.hadoop.fs.TestHDFSFileContextMainOperations



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12767620/HDFS-9198-trunk.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 9cb5d35
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13083/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13083/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13083/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13083/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13083/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 20m 31s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 6 new or modified test files. +1 javac 9m 31s There were no new javac warning messages. +1 javadoc 11m 25s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 33s The applied patch generated 6 new checkstyle issues (total was 414, now 415). +1 whitespace 0m 8s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 41s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 34s Pre-build of native portion -1 hdfs tests 66m 27s Tests failed in hadoop-hdfs.     118m 30s   Reason Tests Failed unit tests hadoop.fs.loadGenerator.TestLoadGenerator   hadoop.fs.TestFcHdfsCreateMkdir   hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.TestRecoverStripedFile   hadoop.fs.TestSWebHdfsFileContextMainOperations   hadoop.fs.TestUrlStreamHandler   hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot   hadoop.fs.viewfs.TestViewFsAtHdfsRoot   hadoop.fs.permission.TestStickyBit   hadoop.hdfs.web.TestWebHDFS Timed out tests org.apache.hadoop.fs.TestEnhancedByteBufferAccess   org.apache.hadoop.fs.TestUrlStreamHandlerFactory   org.apache.hadoop.fs.TestHDFSFileContextMainOperations Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12767620/HDFS-9198-trunk.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 9cb5d35 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13083/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13083/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13083/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13083/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13083/console This message was automatically generated.
          Hide
          umamaheswararao Uma Maheswara Rao G added a comment - - edited

          Thank Daryn for the Nice work here. This is interesting to me.
          I have just review the patch. Following are my comments:

          1. runBlockOp: how about naming it as runBlockReportOp ?
          2. nit:
            while (namesystem.isRunning()) {
            +        NameNodeMetrics metrics = NameNode.getNameNodeMetrics();
            

            May be we can take metrics outside loop and use it?

          3. I think we need to handle throwable for this BR processing thread? incase of any unexpected errors, this thread should not die silently as its one of the important processing thread… ? we may have to terminate the system in such cases.
            minor suggestion: method names in BM could be like runBlockReportOpSync and runBlockReportAsync ?
          4. code format missed for this lines:
            metrics.setBlockOpsQueued(queue.size()+1);
            metrics.addBlockOpsBatched(processed-1);
            
          5. Currently DN sets the flag to trigger sendImmediateIBR on failure of IBR processing. But now we handle Exceptions as NN itself and can not pass to DN as due to async. So now we sendImmdeiateIBR happens only for IPC level exceptions. Have you thought about it. Missing such info would have to wait until next BR right?
          6. Tests looking great to me. minor suggestion is could you please add javadoc for tests?
          Show
          umamaheswararao Uma Maheswara Rao G added a comment - - edited Thank Daryn for the Nice work here. This is interesting to me. I have just review the patch. Following are my comments: runBlockOp: how about naming it as runBlockReportOp ? nit: while (namesystem.isRunning()) { + NameNodeMetrics metrics = NameNode.getNameNodeMetrics(); May be we can take metrics outside loop and use it? I think we need to handle throwable for this BR processing thread? incase of any unexpected errors, this thread should not die silently as its one of the important processing thread… ? we may have to terminate the system in such cases. minor suggestion: method names in BM could be like runBlockReportOpSync and runBlockReportAsync ? code format missed for this lines: metrics.setBlockOpsQueued(queue.size()+1); metrics.addBlockOpsBatched(processed-1); Currently DN sets the flag to trigger sendImmediateIBR on failure of IBR processing. But now we handle Exceptions as NN itself and can not pass to DN as due to async. So now we sendImmdeiateIBR happens only for IPC level exceptions. Have you thought about it. Missing such info would have to wait until next BR right? Tests looking great to me. minor suggestion is could you please add javadoc for tests?
          Hide
          umamaheswararao Uma Maheswara Rao G added a comment -

          Ok, I have just looked your first comments which is trying to answer #5

          Show
          umamaheswararao Uma Maheswara Rao G added a comment - Ok, I have just looked your first comments which is trying to answer #5
          Hide
          daryn Daryn Sharp added a comment -
          1. I named it more generically since it can be used for more, but I'll rename if you feel strongly.
          2. I originally had it outside the loop but something failed. Probably a test. Since it's a class static it should be cheap.
          3. The NN is terminated when the queue stops accepting offers and the thread isn't running, but I'll add a try/catch around the run() loop too.
          4. Will look at the style again.
          5. Yes, I compensated for async IBR failures by making the next heartbeat trigger a re-registration. The only exception we've seen from an IBR is dead/unregistered node so the new code is a no-op but I added it as a safety net.
          6. Ok.
          Show
          daryn Daryn Sharp added a comment - I named it more generically since it can be used for more, but I'll rename if you feel strongly. I originally had it outside the loop but something failed. Probably a test. Since it's a class static it should be cheap. The NN is terminated when the queue stops accepting offers and the thread isn't running, but I'll add a try/catch around the run() loop too. Will look at the style again. Yes, I compensated for async IBR failures by making the next heartbeat trigger a re-registration. The only exception we've seen from an IBR is dead/unregistered node so the new code is a no-op but I added it as a safety net. Ok.
          Hide
          daryn Daryn Sharp added a comment -

          updated per comments. will update branch-2 if this is acceptable.

          Show
          daryn Daryn Sharp added a comment - updated per comments. will update branch-2 if this is acceptable.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 7s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          +1 mvninstall 3m 12s trunk passed
          +1 compile 0m 38s trunk passed with JDK v1.8.0_60
          +1 compile 0m 34s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 17s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          -1 findbugs 2m 3s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs
          +1 javadoc 1m 12s trunk passed with JDK v1.8.0_60
          +1 javadoc 1m 56s trunk passed with JDK v1.7.0_79
          +1 mvninstall 0m 41s the patch passed
          +1 compile 0m 37s the patch passed with JDK v1.8.0_60
          +1 javac 0m 37s the patch passed
          +1 compile 0m 34s the patch passed with JDK v1.7.0_79
          +1 javac 0m 34s the patch passed
          -1 checkstyle 0m 16s Patch generated 5 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 410, now 410).
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 9s the patch passed
          +1 javadoc 1m 14s the patch passed with JDK v1.8.0_60
          +1 javadoc 1m 59s the patch passed with JDK v1.7.0_79
          -1 unit 62m 1s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
          -1 unit 60m 45s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          -1 asflicense 0m 20s Patch generated 58 ASF License warnings.
          143m 55s



          Reason Tests
          JDK v1.8.0_60 Failed junit tests hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
            hadoop.hdfs.server.namenode.TestCacheDirectives
            hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.hdfs.TestDFSClientRetries
            hadoop.hdfs.server.datanode.TestBlockScanner
          JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
            hadoop.hdfs.server.datanode.TestBlockScanner



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-03
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770339/HDFS-9198-trunk.patch
          JIRA Issue HDFS-9198
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux c82d40790283 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-1a9afee/precommit/personality/hadoop.sh
          git revision trunk / 957f031
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13359/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13359/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13359/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13359/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13359/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13359/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13359/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13359/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 226MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13359/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 7s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 3m 12s trunk passed +1 compile 0m 38s trunk passed with JDK v1.8.0_60 +1 compile 0m 34s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 17s trunk passed +1 mvneclipse 0m 14s trunk passed -1 findbugs 2m 3s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs +1 javadoc 1m 12s trunk passed with JDK v1.8.0_60 +1 javadoc 1m 56s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 41s the patch passed +1 compile 0m 37s the patch passed with JDK v1.8.0_60 +1 javac 0m 37s the patch passed +1 compile 0m 34s the patch passed with JDK v1.7.0_79 +1 javac 0m 34s the patch passed -1 checkstyle 0m 16s Patch generated 5 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 410, now 410). +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 9s the patch passed +1 javadoc 1m 14s the patch passed with JDK v1.8.0_60 +1 javadoc 1m 59s the patch passed with JDK v1.7.0_79 -1 unit 62m 1s hadoop-hdfs in the patch failed with JDK v1.8.0_60. -1 unit 60m 45s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 20s Patch generated 58 ASF License warnings. 143m 55s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.server.namenode.TestCacheDirectives   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.datanode.TestBlockScanner JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.server.datanode.TestBlockScanner Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-03 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770339/HDFS-9198-trunk.patch JIRA Issue HDFS-9198 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux c82d40790283 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-1a9afee/precommit/personality/hadoop.sh git revision trunk / 957f031 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13359/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13359/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13359/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13359/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13359/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13359/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13359/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13359/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 226MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13359/console This message was automatically generated.
          Hide
          daryn Daryn Sharp added a comment -

          Comments? Been running successfully with this feature since early Nov. Will update patch if necessary if it's currently acceptable. Seem to recall test failures were general build instability.

          Show
          daryn Daryn Sharp added a comment - Comments? Been running successfully with this feature since early Nov. Will update patch if necessary if it's currently acceptable. Seem to recall test failures were general build instability.
          Hide
          umamaheswararao Uma Maheswara Rao G added a comment -

          Hi Daryn, Apologies for the delay on this. Let me finish it today my final pass on this. Thanks.

          Show
          umamaheswararao Uma Maheswara Rao G added a comment - Hi Daryn, Apologies for the delay on this. Let me finish it today my final pass on this. Thanks.
          Hide
          umamaheswararao Uma Maheswara Rao G added a comment -

          Daryn Sharp, Latest patch looks good to me. Could you please update patch on the latest trunk code. Seems it is failing to apply cleanly. Sorry for the in time review on this. Also it will get chance to run the jenkins on latest code base. Thanks

          Show
          umamaheswararao Uma Maheswara Rao G added a comment - Daryn Sharp , Latest patch looks good to me. Could you please update patch on the latest trunk code. Seems it is failing to apply cleanly. Sorry for the in time review on this. Also it will get chance to run the jenkins on latest code base. Thanks
          Hide
          daryn Daryn Sharp added a comment -

          No functional change, just updated to account for line number drift.

          Show
          daryn Daryn Sharp added a comment - No functional change, just updated to account for line number drift.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          +1 mvninstall 8m 39s trunk passed
          +1 compile 0m 51s trunk passed with JDK v1.8.0_66
          +1 compile 0m 47s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 19s trunk passed
          +1 mvnsite 1m 0s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 2m 6s trunk passed
          +1 javadoc 1m 16s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 59s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 57s the patch passed
          +1 compile 0m 53s the patch passed with JDK v1.8.0_66
          -1 javac 7m 13s hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 32, now 32).
          +1 javac 0m 53s the patch passed
          +1 compile 0m 54s the patch passed with JDK v1.7.0_91
          -1 javac 8m 9s hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91 with JDK v1.7.0_91 generated 3 new issues (was 34, now 34).
          +1 javac 0m 54s the patch passed
          -1 checkstyle 0m 20s Patch generated 5 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 384, now 384).
          +1 mvnsite 1m 7s the patch passed
          +1 mvneclipse 0m 17s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 31s the patch passed
          +1 javadoc 1m 24s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 11s the patch passed with JDK v1.7.0_91
          -1 unit 66m 43s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 57m 48s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 25s Patch generated 58 ASF License warnings.
          155m 53s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.datanode.TestRefreshNamenodes
            hadoop.hdfs.web.TestWebHDFS
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.server.namenode.TestNNThroughputBenchmark



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778095/HDFS-9198-trunk.patch
          JIRA Issue HDFS-9198
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c075728831b6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0f708d4
          findbugs v3.0.0
          javac hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13902/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          javac hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13902/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13902/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13902/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13902/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13902/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13902/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13902/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13902/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 75MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13902/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 8m 39s trunk passed +1 compile 0m 51s trunk passed with JDK v1.8.0_66 +1 compile 0m 47s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 19s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 6s trunk passed +1 javadoc 1m 16s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 59s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 57s the patch passed +1 compile 0m 53s the patch passed with JDK v1.8.0_66 -1 javac 7m 13s hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 32, now 32). +1 javac 0m 53s the patch passed +1 compile 0m 54s the patch passed with JDK v1.7.0_91 -1 javac 8m 9s hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91 with JDK v1.7.0_91 generated 3 new issues (was 34, now 34). +1 javac 0m 54s the patch passed -1 checkstyle 0m 20s Patch generated 5 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 384, now 384). +1 mvnsite 1m 7s the patch passed +1 mvneclipse 0m 17s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 31s the patch passed +1 javadoc 1m 24s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 11s the patch passed with JDK v1.7.0_91 -1 unit 66m 43s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 57m 48s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 25s Patch generated 58 ASF License warnings. 155m 53s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.datanode.TestRefreshNamenodes   hadoop.hdfs.web.TestWebHDFS   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778095/HDFS-9198-trunk.patch JIRA Issue HDFS-9198 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c075728831b6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0f708d4 findbugs v3.0.0 javac hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13902/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt javac hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13902/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13902/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13902/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13902/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13902/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13902/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13902/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13902/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13902/console This message was automatically generated.
          Hide
          umamaheswararao Uma Maheswara Rao G added a comment -

          +1 on the latest patch. Failures seems unrelated. Committing.

          Show
          umamaheswararao Uma Maheswara Rao G added a comment - +1 on the latest patch. Failures seems unrelated. Committing.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8982 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8982/)
          HDFS-9198. Coalesce IBR processing in the NN. (Daryn Sharp via (umamahesh: rev f741476146574550a1a208d58ef8be76639e5ddc)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/BlockReportTestBase.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeRegistration.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBrVariations.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingReplication.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8982 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8982/ ) HDFS-9198 . Coalesce IBR processing in the NN. (Daryn Sharp via (umamahesh: rev f741476146574550a1a208d58ef8be76639e5ddc) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/BlockReportTestBase.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeRegistration.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBrVariations.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingReplication.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java
          Hide
          umamaheswararao Uma Maheswara Rao G added a comment -

          Thanks Daryn Sharp for the nice work here. I have committed this to trunk, branch-2 and branch-2.8

          Show
          umamaheswararao Uma Maheswara Rao G added a comment - Thanks Daryn Sharp for the nice work here. I have committed this to trunk, branch-2 and branch-2.8
          Hide
          umamaheswararao Uma Maheswara Rao G added a comment -

          While merging to branch-2, I did the following edits to resolve conflicts.

          -  DatanodeStorageInfo[] getStorageInfos() {
          +  @VisibleForTesting
          +  public DatanodeStorageInfo[] getStorageInfos() {
          

          Test depending on this. This change already exist in trunk, but not in branch-2, just made it visible to testcode. Attached the patches what I committed to branch-2 and 2.8 for reference.

          Show
          umamaheswararao Uma Maheswara Rao G added a comment - While merging to branch-2, I did the following edits to resolve conflicts. - DatanodeStorageInfo[] getStorageInfos() { + @VisibleForTesting + public DatanodeStorageInfo[] getStorageInfos() { Test depending on this. This change already exist in trunk, but not in branch-2, just made it visible to testcode. Attached the patches what I committed to branch-2 and 2.8 for reference.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Daryn Sharp, Is it worth merging this to 2.7.3?

          Show
          vinayrpet Vinayakumar B added a comment - Daryn Sharp , Is it worth merging this to 2.7.3?
          Hide
          kihwal Kihwal Lee added a comment -

          We have applied this internally to 2.7 and run for quite some time. So I would say it is safe. HDFS-8999 and HDFS-9710 will make it less important, but since these are more invasive and risky, HDFS-9198 in 2.7 might be a better alternative for now.

          Show
          kihwal Kihwal Lee added a comment - We have applied this internally to 2.7 and run for quite some time. So I would say it is safe. HDFS-8999 and HDFS-9710 will make it less important, but since these are more invasive and risky, HDFS-9198 in 2.7 might be a better alternative for now.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Here is the branch-2.7 patch, resolved some conflicts and compilation errors.

          Show
          vinayrpet Vinayakumar B added a comment - Here is the branch-2.7 patch, resolved some conflicts and compilation errors.
          Hide
          kihwal Kihwal Lee added a comment -

          +1 for the branch-2.7 patch. It is code-wise identical to what we are running internally.

          Show
          kihwal Kihwal Lee added a comment - +1 for the branch-2.7 patch. It is code-wise identical to what we are running internally.
          Hide
          kihwal Kihwal Lee added a comment -

          Relevant tests are all passing with this patch.

          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.hdfs.server.datanode.TestIncrementalBrVariations
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.301 sec - in org.apache.hadoop.hdfs.server.datanode.TestIncrementalBrVariations
          OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.83 sec - in org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode
          OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager
          Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.238 sec - in org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager
          OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.hdfs.server.blockmanagement.TestPendingReplication
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.094 sec - in org.apache.hadoop.hdfs.server.blockmanagement.TestPendingReplication
          OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
          Running org.apache.hadoop.hdfs.TestDatanodeRegistration
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.187 sec - in org.apache.hadoop.hdfs.TestDatanodeRegistration
          
          Results :
          
          Tests run: 32, Failures: 0, Errors: 0, Skipped: 0
          

          Just committed to branch-2.7. Thanks, Vinay!

          Show
          kihwal Kihwal Lee added a comment - Relevant tests are all passing with this patch. ------------------------------------------------------- T E S T S ------------------------------------------------------- OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.server.datanode.TestIncrementalBrVariations Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.301 sec - in org.apache.hadoop.hdfs.server.datanode.TestIncrementalBrVariations OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.83 sec - in org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.238 sec - in org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.server.blockmanagement.TestPendingReplication Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.094 sec - in org.apache.hadoop.hdfs.server.blockmanagement.TestPendingReplication OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.TestDatanodeRegistration Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.187 sec - in org.apache.hadoop.hdfs.TestDatanodeRegistration Results : Tests run: 32, Failures: 0, Errors: 0, Skipped: 0 Just committed to branch-2.7. Thanks, Vinay!
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.

            People

            • Assignee:
              daryn Daryn Sharp
              Reporter:
              daryn Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              31 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development