Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12189

Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: ipc, test
    • Labels:
      None
    • Target Version/s:

      Description

      Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible. This is the trade-off between performance and functionality, even in the very very rare situation, we may drop one element, but it is not the end of the world since the client may still recover with timeout.
      CallQueueManager may drop elements from the queue sometimes when calling swapQueue.
      The following test failure from TestCallQueueManager shown some elements in the queue are dropped.
      https://builds.apache.org/job/PreCommit-HADOOP-Build/7150/testReport/org.apache.hadoop.ipc/TestCallQueueManager/testSwapUnderContention/

      java.lang.AssertionError: expected:<27241> but was:<27245>
      	at org.junit.Assert.fail(Assert.java:88)
      	at org.junit.Assert.failNotEquals(Assert.java:743)
      	at org.junit.Assert.assertEquals(Assert.java:118)
      	at org.junit.Assert.assertEquals(Assert.java:555)
      	at org.junit.Assert.assertEquals(Assert.java:542)
      	at org.apache.hadoop.ipc.TestCallQueueManager.testSwapUnderContention(TestCallQueueManager.java:220)
      

      It looked like the elements in the queue are dropped due to CallQueueManager#swapQueue
      Looked at the implementation of CallQueueManager#swapQueue, there is a possibility that the elements in the queue are dropped. If the queue is full, the calling thread for CallQueueManager#put is blocked for long time. It may put the element into the old queue after queue in takeRef is changed by swapQueue, then this element in the old queue will be dropped.

      1. HADOOP-12189.none_guarantee.002.patch
        3 kB
        zhihai xu
      2. HADOOP-12189.none_guarantee.001.patch
        3 kB
        zhihai xu
      3. HADOOP-12189.none_guarantee.000.patch
        7 kB
        zhihai xu
      4. HADOOP-12189.001.patch
        7 kB
        zhihai xu
      5. HADOOP-12189.000.patch
        6 kB
        zhihai xu

        Activity

        Hide
        zxu zhihai xu added a comment -

        I uploaded a patch HADOOP-12189.000.patch for review. This patch will wait for the threads blocked by CallQueueManager#put in swapQueue, so it will make sure all the elements from old queue added by these blocked threads are processed.

        Show
        zxu zhihai xu added a comment - I uploaded a patch HADOOP-12189 .000.patch for review. This patch will wait for the threads blocked by CallQueueManager#put in swapQueue, so it will make sure all the elements from old queue added by these blocked threads are processed.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 29s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 32s There were no new javac warning messages.
        +1 javadoc 9m 36s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 4s The applied patch generated 1 new checkstyle issues (total was 6, now 7).
        +1 whitespace 0m 1s The patch has no lines that end in whitespace.
        +1 install 1m 33s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 1m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 common tests 21m 54s Tests passed in hadoop-common.
            60m 59s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12743652/HADOOP-12189.000.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 688617d
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7152/artifact/patchprocess/diffcheckstylehadoop-common.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7152/artifact/patchprocess/testrun_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7152/testReport/
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7152/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 29s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 32s There were no new javac warning messages. +1 javadoc 9m 36s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 4s The applied patch generated 1 new checkstyle issues (total was 6, now 7). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 21m 54s Tests passed in hadoop-common.     60m 59s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12743652/HADOOP-12189.000.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 688617d checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7152/artifact/patchprocess/diffcheckstylehadoop-common.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7152/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7152/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7152/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 7s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 28s There were no new javac warning messages.
        +1 javadoc 9m 33s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 4s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 1m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 common tests 21m 56s Tests passed in hadoop-common.
            60m 31s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12743657/HADOOP-12189.000.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 688617d
        hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7153/artifact/patchprocess/testrun_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7153/testReport/
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7153/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 7s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 28s There were no new javac warning messages. +1 javadoc 9m 33s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 4s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 21m 56s Tests passed in hadoop-common.     60m 31s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12743657/HADOOP-12189.000.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 688617d hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7153/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7153/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7153/console This message was automatically generated.
        Hide
        zxu zhihai xu added a comment -

        The patch also changed the test to make this issue happen much easier, without the change, the test will fail.

        Show
        zxu zhihai xu added a comment - The patch also changed the test to make this issue happen much easier, without the change, the test will fail.
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 33s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 37s There were no new javac warning messages.
        +1 javadoc 9m 45s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 4s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 35s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 1m 53s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 common tests 22m 0s Tests passed in hadoop-common.
            61m 25s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12743661/HADOOP-12189.000.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 688617d
        hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7154/artifact/patchprocess/testrun_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7154/testReport/
        Java 1.7.0_55
        uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7154/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 33s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 37s There were no new javac warning messages. +1 javadoc 9m 45s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 4s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 53s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 0s Tests passed in hadoop-common.     61m 25s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12743661/HADOOP-12189.000.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 688617d hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7154/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7154/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7154/console This message was automatically generated.
        Hide
        kihwal Kihwal Lee added a comment -

        Chris Li, Ming Ma, would you take a look at this?

        Show
        kihwal Kihwal Lee added a comment - Chris Li , Ming Ma , would you take a look at this?
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 15m 5s Findbugs (version ) appears to be broken on trunk.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 33s There were no new javac warning messages.
        +1 javadoc 9m 36s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 39s There were no new checkstyle issues.
        +1 whitespace 0m 1s The patch has no lines that end in whitespace.
        +1 install 1m 32s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 1m 49s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 common tests 22m 8s Tests passed in hadoop-common.
            59m 21s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12743757/HADOOP-12189.000.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / fc92d3e
        hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7162/artifact/patchprocess/testrun_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7162/testReport/
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7162/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 15m 5s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 33s There were no new javac warning messages. +1 javadoc 9m 36s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 39s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 49s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 8s Tests passed in hadoop-common.     59m 21s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12743757/HADOOP-12189.000.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / fc92d3e hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7162/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7162/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7162/console This message was automatically generated.
        Hide
        chrilisf Chris Li added a comment -

        zhihai xu One of the design decisions of queue swapping is that it not require locking for performance reasons. A tradeoff was made where calls could be dropped in rare occasions, but this should be super rare, like a theoretical concern more than a practical one, so if you're seeing dropped calls we should fix that.

        However, last we discussed, this code should be lock free, so we can't depend on using synchronized() blocks.

        Also it seems that this code doesn't provide 100% guarantee either, since the queue can still be swapped in between bq = putRef.get() and num.incrementAndGet()

        See https://issues.apache.org/jira/browse/HADOOP-10278

        Show
        chrilisf Chris Li added a comment - zhihai xu One of the design decisions of queue swapping is that it not require locking for performance reasons. A tradeoff was made where calls could be dropped in rare occasions, but this should be super rare, like a theoretical concern more than a practical one, so if you're seeing dropped calls we should fix that. However, last we discussed, this code should be lock free, so we can't depend on using synchronized() blocks. Also it seems that this code doesn't provide 100% guarantee either, since the queue can still be swapped in between bq = putRef.get() and num.incrementAndGet() See https://issues.apache.org/jira/browse/HADOOP-10278
        Hide
        zxu zhihai xu added a comment -

        Thanks for the review Chris Li!

        but this should be super rare, like a theoretical concern more than a practical one, so if you're seeing dropped calls we should fix that.

        Yes, it may rarely happen but the original test will fail Intermittently, The updated unit test can reproduce this problem easily.

        this code should be lock free, so we can't depend on using synchronized() blocks.

        Yes, there will be some performance degradation. But the code inside the lock is very small. the performance overhead with the patch is not much. So it will depend on what is the impact if we drop the elements in the queue.

        synchronized (queueMap) {
        bq = putRef.get();
        num = queueMap.get(bq);
        num.incrementAndGet();
        }
        

        Also it seems that this code doesn't provide 100% guarantee either, since the queue can still be swapped in between bq = putRef.get() and num.incrementAndGet()

        The patch can handle the above situation. If the queue is swapped between bq = putRef.get() and num.incrementAndGet(), bq will be oldQ, because of the lock for synchronizedMap queueMap, queueMap.remove(oldQ); can only be called after num.incrementAndGet(). This will make sure swapQueue see the updated num. So I think the patch can guarantee the elements added by put won't be dropped.

        Show
        zxu zhihai xu added a comment - Thanks for the review Chris Li ! but this should be super rare, like a theoretical concern more than a practical one, so if you're seeing dropped calls we should fix that. Yes, it may rarely happen but the original test will fail Intermittently, The updated unit test can reproduce this problem easily. this code should be lock free, so we can't depend on using synchronized() blocks. Yes, there will be some performance degradation. But the code inside the lock is very small. the performance overhead with the patch is not much. So it will depend on what is the impact if we drop the elements in the queue. synchronized (queueMap) { bq = putRef.get(); num = queueMap.get(bq); num.incrementAndGet(); } Also it seems that this code doesn't provide 100% guarantee either, since the queue can still be swapped in between bq = putRef.get() and num.incrementAndGet() The patch can handle the above situation. If the queue is swapped between bq = putRef.get() and num.incrementAndGet() , bq will be oldQ, because of the lock for synchronizedMap queueMap , queueMap.remove(oldQ); can only be called after num.incrementAndGet() . This will make sure swapQueue see the updated num . So I think the patch can guarantee the elements added by put won't be dropped.
        Hide
        chrilisf Chris Li added a comment -

        Hi zhihai xu, I see, it's a synchronized map, so its methods acquire the queue's intrinsic lock. Thanks for clarifying

        I have concerns with performance. Have you measured impact? Last I checked, acquiring a lock every time resulted in measurable performance penalty. It's not just time under contention, it's also the fact that synchronized will introduce a memory barrier, and having 120 threads contesting over yet another lock when processing 60k requests per second sounds questionable.

        I think it all comes down to performance. The unsafe "sleep 10ms and try again" scheme was born from that (since queue swapping is a relatively infrequent event compared to put(). If the performance situation has changed since we last discussed this, then we can come up with better solutions that are guarantee consistency.

        So options are:
        1. Start locking and provide strict consistency (and then we don't need to Thread.sleep)
        2. Try to increase sleep timeout or increase number of checks to make call loss practically impossible
        3. Relax tests and accept the loss of calls during queue swaps

        Daryn Sharp might have some thoughts.

        Show
        chrilisf Chris Li added a comment - Hi zhihai xu , I see, it's a synchronized map, so its methods acquire the queue's intrinsic lock. Thanks for clarifying I have concerns with performance. Have you measured impact? Last I checked, acquiring a lock every time resulted in measurable performance penalty. It's not just time under contention, it's also the fact that synchronized will introduce a memory barrier, and having 120 threads contesting over yet another lock when processing 60k requests per second sounds questionable. I think it all comes down to performance. The unsafe "sleep 10ms and try again" scheme was born from that (since queue swapping is a relatively infrequent event compared to put() . If the performance situation has changed since we last discussed this, then we can come up with better solutions that are guarantee consistency. So options are: 1. Start locking and provide strict consistency (and then we don't need to Thread.sleep) 2. Try to increase sleep timeout or increase number of checks to make call loss practically impossible 3. Relax tests and accept the loss of calls during queue swaps Daryn Sharp might have some thoughts.
        Hide
        zxu zhihai xu added a comment -

        I just changed the unit test to use 120 threads for put and 120 threads for take without calling swapQueue.
        With patch, it can process 260,310 elements in less than 293ms.
        Without patch, it can process 437,208 elements in less than 289ms.
        So it looks like 60,000 requests per second with 120 threads may be supported with the patch.
        Yes, increasing number of check points may decrease the loss chance.
        Yes, for lossless swapQueue, we can move Thread.sleep from queueIsReallyEmpty into swapQueue and only check one point in queueIsReallyEmpty.

        Show
        zxu zhihai xu added a comment - I just changed the unit test to use 120 threads for put and 120 threads for take without calling swapQueue. With patch, it can process 260,310 elements in less than 293ms. Without patch, it can process 437,208 elements in less than 289ms. So it looks like 60,000 requests per second with 120 threads may be supported with the patch. Yes, increasing number of check points may decrease the loss chance. Yes, for lossless swapQueue, we can move Thread.sleep from queueIsReallyEmpty into swapQueue and only check one point in queueIsReallyEmpty.
        Hide
        chrilisf Chris Li added a comment -

        Wait, did the throughput drop from 437k/300ms to 260k/300ms? That's a pretty big drop. Even if it's not the bottleneck in the server, it's still extra overhead. Let's wait for Daryn's thoughts, because he guided the original design

        Show
        chrilisf Chris Li added a comment - Wait, did the throughput drop from 437k/300ms to 260k/300ms? That's a pretty big drop. Even if it's not the bottleneck in the server, it's still extra overhead. Let's wait for Daryn's thoughts, because he guided the original design
        Hide
        zxu zhihai xu added a comment -

        But if I use 1 thread for put and 1 thread for take, they are very close, less than 10% difference.
        With patch, it can process 472,669 elements in less than 254ms.
        Without patch, it can process 489,810 elements in less than 253ms.
        So the lock will affect the performance with more threads.

        Show
        zxu zhihai xu added a comment - But if I use 1 thread for put and 1 thread for take, they are very close, less than 10% difference. With patch, it can process 472,669 elements in less than 254ms. Without patch, it can process 489,810 elements in less than 253ms. So the lock will affect the performance with more threads.
        Hide
        zxu zhihai xu added a comment -

        I uploaded a new patch HADOOP-12189.001.patch. I did some optimization in the new patch.
        With the new patch, the performance is much better. it can process 377,932 elements in less than 297ms using 120 threads for put and 120 threads for take. The performance with the new patch is very close to the performance without patch, less than 15% difference.

        Show
        zxu zhihai xu added a comment - I uploaded a new patch HADOOP-12189 .001.patch. I did some optimization in the new patch. With the new patch, the performance is much better. it can process 377,932 elements in less than 297ms using 120 threads for put and 120 threads for take. The performance with the new patch is very close to the performance without patch, less than 15% difference.
        Hide
        chrilisf Chris Li added a comment -

        That's a significant performance hit! I think we should aim for a lock-free solution in the <1% penalty range, hopefully nothing too complex either for code maintainability.

        Show
        chrilisf Chris Li added a comment - That's a significant performance hit! I think we should aim for a lock-free solution in the <1% penalty range, hopefully nothing too complex either for code maintainability.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 20m 52s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 9m 47s There were no new javac warning messages.
        +1 javadoc 12m 38s There were no new javadoc warning messages.
        +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 22s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 57s mvn install still works.
        +1 eclipse:eclipse 0m 40s The patch built with eclipse:eclipse.
        -1 findbugs 2m 21s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
        +1 common tests 24m 42s Tests passed in hadoop-common.
            74m 50s  



        Reason Tests
        FindBugs module:hadoop-common



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12744033/HADOOP-12189.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 3dc92e8
        Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/7182/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
        hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7182/artifact/patchprocess/testrun_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7182/testReport/
        Java 1.7.0_55
        uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7182/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 20m 52s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 9m 47s There were no new javac warning messages. +1 javadoc 12m 38s There were no new javadoc warning messages. +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 22s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 57s mvn install still works. +1 eclipse:eclipse 0m 40s The patch built with eclipse:eclipse. -1 findbugs 2m 21s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 common tests 24m 42s Tests passed in hadoop-common.     74m 50s   Reason Tests FindBugs module:hadoop-common Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744033/HADOOP-12189.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 3dc92e8 Findbugs warnings https://builds.apache.org/job/PreCommit-HADOOP-Build/7182/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7182/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7182/testReport/ Java 1.7.0_55 uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7182/console This message was automatically generated.
        Hide
        zxu zhihai xu added a comment -

        I looked at HADOOP-10278, It looks like we don't want to add any locking during normal operation.
        So it looks like increasing number of checkpoint may be an alternative solution, which is based on Chris Li's suggestion.
        I uploaded a new patch HADOOP-12189.none_guarantee.000.patch, which make the number and interval of checkpoint configurable.
        The default number of checkpoint will be increased to 4 from 2. I run the test multiple times, I didn't see any failure with the patch. Without the patch, it almost failed every time.

        Show
        zxu zhihai xu added a comment - I looked at HADOOP-10278 , It looks like we don't want to add any locking during normal operation. So it looks like increasing number of checkpoint may be an alternative solution, which is based on Chris Li 's suggestion. I uploaded a new patch HADOOP-12189 .none_guarantee.000.patch, which make the number and interval of checkpoint configurable. The default number of checkpoint will be increased to 4 from 2. I run the test multiple times, I didn't see any failure with the patch. Without the patch, it almost failed every time.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 18s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 30s There were no new javac warning messages.
        +1 javadoc 9m 36s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 4s The applied patch generated 2 new checkstyle issues (total was 81, now 83).
        +1 whitespace 0m 1s The patch has no lines that end in whitespace.
        +1 install 1m 31s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 1m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        -1 common tests 21m 40s Tests failed in hadoop-common.
            60m 28s  



        Reason Tests
        Failed unit tests hadoop.ipc.TestCallQueueManager



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12744147/HADOOP-12189.none_guarantee.000.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / c9dd2ca
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7191/artifact/patchprocess/diffcheckstylehadoop-common.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7191/artifact/patchprocess/testrun_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7191/testReport/
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7191/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 18s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 36s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 4s The applied patch generated 2 new checkstyle issues (total was 81, now 83). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 21m 40s Tests failed in hadoop-common.     60m 28s   Reason Tests Failed unit tests hadoop.ipc.TestCallQueueManager Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744147/HADOOP-12189.none_guarantee.000.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c9dd2ca checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7191/artifact/patchprocess/diffcheckstylehadoop-common.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7191/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7191/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7191/console This message was automatically generated.
        Hide
        zxu zhihai xu added a comment -

        It looks like TestCallQueueManager is failed in Jenkins although it passed in my local test. Updated the patch by increasing default checkpointNum to 8.

        Show
        zxu zhihai xu added a comment - It looks like TestCallQueueManager is failed in Jenkins although it passed in my local test. Updated the patch by increasing default checkpointNum to 8.
        Hide
        arpitagarwal Arpit Agarwal added a comment - - edited

        Hi zhihai xu, I am also not sure this issue needs fixing. Dropping some requests in the rare event of a queue swap looks harmless. I assume clients will experience timeout exceptions and retry.

        Show
        arpitagarwal Arpit Agarwal added a comment - - edited Hi zhihai xu , I am also not sure this issue needs fixing. Dropping some requests in the rare event of a queue swap looks harmless. I assume clients will experience timeout exceptions and retry.
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 13s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 33s There were no new javac warning messages.
        +1 javadoc 9m 39s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 3s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 19s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 1m 49s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 common tests 22m 14s Tests passed in hadoop-common.
            60m 49s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12744292/HADOOP-12189.none_guarantee.000.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 625d7ed
        hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7201/artifact/patchprocess/testrun_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7201/testReport/
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7201/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 13s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 33s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 3s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 19s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 49s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 14s Tests passed in hadoop-common.     60m 49s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744292/HADOOP-12189.none_guarantee.000.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 625d7ed hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7201/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7201/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7201/console This message was automatically generated.
        Hide
        chrilisf Chris Li added a comment -

        Arpit Agarwal I think zhihai xu encountered unit test failures, which brought his attention here.

        If increasing checkpoints works then we should do that. I don't think we should introduce a new config parameter though... this is something nobody will ever modify and controls such a low level detail. I'd suggest experimenting with what passes and then increasing the tolerance by an order of magnitude to be safe... so if you can get it to pass with 10 checks at 2ms pause, then we can do 20-100 checks at 2ms pause (as long as the total wait time is < 1 second). This is basically mostly for developer's sakes, since as Arpit mentioned, dropping is a rarity even during queue swaps in real life.

        Show
        chrilisf Chris Li added a comment - Arpit Agarwal I think zhihai xu encountered unit test failures, which brought his attention here. If increasing checkpoints works then we should do that. I don't think we should introduce a new config parameter though... this is something nobody will ever modify and controls such a low level detail. I'd suggest experimenting with what passes and then increasing the tolerance by an order of magnitude to be safe... so if you can get it to pass with 10 checks at 2ms pause, then we can do 20-100 checks at 2ms pause (as long as the total wait time is < 1 second). This is basically mostly for developer's sakes, since as Arpit mentioned, dropping is a rarity even during queue swaps in real life.
        Hide
        zxu zhihai xu added a comment -

        Arpit Agarwal, thanks for the valuable information. It is good to know the clients can still recover with timeout and retry even though the requests are dropped by server. In this case, decreasing the chance of dropping queue elements without affecting the performance for normal operations may be enough.
        Chris Li, thanks for the review and suggestion. I uploaded a new patch HADOOP-12189.none_guarantee.001.patch, which addressed your comments. I increase the number of checkpoint to 20 which gives us more margin, 200(20*10)ms extra waiting time is much less than 1 second. I also fixed a typo in the new patch. Please review it.

        Show
        zxu zhihai xu added a comment - Arpit Agarwal , thanks for the valuable information. It is good to know the clients can still recover with timeout and retry even though the requests are dropped by server. In this case, decreasing the chance of dropping queue elements without affecting the performance for normal operations may be enough. Chris Li , thanks for the review and suggestion. I uploaded a new patch HADOOP-12189 .none_guarantee.001.patch, which addressed your comments. I increase the number of checkpoint to 20 which gives us more margin, 200(20*10)ms extra waiting time is much less than 1 second. I also fixed a typo in the new patch. Please review it.
        Hide
        chrilisf Chris Li added a comment -

        Great, couple more minor things:

        Can we update the comment for queueReallyIsEmpty, and also include how the constants CHECKPOINT_INTERVAL_MS and CHECKPOINT_NUM come into play?

        What do you think about expressing the code in a for loop? Makes the code more readable I think:

        private boolean queueIsReallyEmpty(BlockingQueue<?> q) {
          for (int i = 0; i < CHECKPOINT_NUM; i++) {
            if (!q.isEmpty()) {
              return false;
            }
        
            try {
              Thread.sleep(CHECKPOINT_INTERVAL_MS);
            } catch (InterruptedException ie) {
              return false;
            }
          }
        
          return true;
        }
        
        
        Show
        chrilisf Chris Li added a comment - Great, couple more minor things: Can we update the comment for queueReallyIsEmpty, and also include how the constants CHECKPOINT_INTERVAL_MS and CHECKPOINT_NUM come into play? What do you think about expressing the code in a for loop? Makes the code more readable I think: private boolean queueIsReallyEmpty(BlockingQueue<?> q) { for ( int i = 0; i < CHECKPOINT_NUM; i++) { if (!q.isEmpty()) { return false ; } try { Thread .sleep(CHECKPOINT_INTERVAL_MS); } catch (InterruptedException ie) { return false ; } } return true ; }
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 17m 45s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 8m 13s There were no new javac warning messages.
        +1 javadoc 10m 2s There were no new javadoc warning messages.
        +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 12s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 26s mvn install still works.
        +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
        +1 findbugs 1m 56s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 common tests 22m 36s Tests passed in hadoop-common.
            64m 10s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12744540/HADOOP-12189.none_guarantee.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / f4ca530
        hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7213/artifact/patchprocess/testrun_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7213/testReport/
        Java 1.7.0_55
        uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7213/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 45s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 13s There were no new javac warning messages. +1 javadoc 10m 2s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 12s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 26s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 1m 56s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 36s Tests passed in hadoop-common.     64m 10s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744540/HADOOP-12189.none_guarantee.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f4ca530 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7213/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7213/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7213/console This message was automatically generated.
        Hide
        zxu zhihai xu added a comment -

        Thanks for the review Chris Li! I use for loop but I check if queue is empty after sleep because I want to avoid continuously checking without delay. I uploaded a new patch HADOOP-12189.none_guarantee.002.patch based on your comment, please review it.

        Show
        zxu zhihai xu added a comment - Thanks for the review Chris Li ! I use for loop but I check if queue is empty after sleep because I want to avoid continuously checking without delay. I uploaded a new patch HADOOP-12189 .none_guarantee.002.patch based on your comment, please review it.
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 17m 0s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 52s There were no new javac warning messages.
        +1 javadoc 9m 47s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 8s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 21s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 1m 51s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 common tests 22m 42s Tests passed in hadoop-common.
            62m 42s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12744581/HADOOP-12189.none_guarantee.002.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / f4ca530
        hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7218/artifact/patchprocess/testrun_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7218/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7218/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 0s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 52s There were no new javac warning messages. +1 javadoc 9m 47s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 8s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 21s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 51s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 42s Tests passed in hadoop-common.     62m 42s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744581/HADOOP-12189.none_guarantee.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f4ca530 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7218/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7218/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7218/console This message was automatically generated.
        Hide
        chrilisf Chris Li added a comment -

        Looks good to me. +1

        Show
        chrilisf Chris Li added a comment - Looks good to me. +1
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 17m 10s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 8m 0s There were no new javac warning messages.
        +1 javadoc 10m 7s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 10s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 22s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 1m 57s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        -1 common tests 22m 34s Tests failed in hadoop-common.
            63m 17s  



        Reason Tests
        Timed out tests org.apache.hadoop.ipc.TestIPC



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12744585/HADOOP-12189.none_guarantee.002.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / f4ca530
        hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7220/artifact/patchprocess/testrun_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7220/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7220/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 10s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 0s There were no new javac warning messages. +1 javadoc 10m 7s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 10s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 22s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 57s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 22m 34s Tests failed in hadoop-common.     63m 17s   Reason Tests Timed out tests org.apache.hadoop.ipc.TestIPC Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744585/HADOOP-12189.none_guarantee.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f4ca530 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7220/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7220/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7220/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 39s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 44s There were no new javac warning messages.
        +1 javadoc 9m 48s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 7s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 21s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 1m 54s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 common tests 22m 23s Tests passed in hadoop-common.
            61m 54s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12744617/HADOOP-12189.none_guarantee.002.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 1a0752d
        hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7223/artifact/patchprocess/testrun_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7223/testReport/
        Java 1.7.0_55
        uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7223/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 39s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 44s There were no new javac warning messages. +1 javadoc 9m 48s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 7s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 21s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 54s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 23s Tests passed in hadoop-common.     61m 54s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744617/HADOOP-12189.none_guarantee.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 1a0752d hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7223/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7223/testReport/ Java 1.7.0_55 uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7223/console This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        LGTM too, will commit shortly. Thanks Zhihai for the patch, Chris and Arpit for reviewing.

        Show
        andrew.wang Andrew Wang added a comment - LGTM too, will commit shortly. Thanks Zhihai for the patch, Chris and Arpit for reviewing.
        Hide
        andrew.wang Andrew Wang added a comment -

        I committed this to branch-2 and trunk, since it seems like essentially a test fix I don't think it needs to go to branch-2.7. Thanks again all.

        Show
        andrew.wang Andrew Wang added a comment - I committed this to branch-2 and trunk, since it seems like essentially a test fix I don't think it needs to go to branch-2.7. Thanks again all.
        Hide
        zxu zhihai xu added a comment -

        Thanks Chris and Arpit for reviewing! thanks Andrew for reviewing and committing the patch!

        Show
        zxu zhihai xu added a comment - Thanks Chris and Arpit for reviewing! thanks Andrew for reviewing and committing the patch!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8208 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8208/)
        HADOOP-12189. Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible. Contributed by Zhihai Xu. (wang: rev 6736a1ab7033523ed5f304fdfed46d7f348665b4)

        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestCallQueueManager.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8208 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8208/ ) HADOOP-12189 . Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible. Contributed by Zhihai Xu. (wang: rev 6736a1ab7033523ed5f304fdfed46d7f348665b4) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestCallQueueManager.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #996 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/996/)
        HADOOP-12189. Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible. Contributed by Zhihai Xu. (wang: rev 6736a1ab7033523ed5f304fdfed46d7f348665b4)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestCallQueueManager.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #996 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/996/ ) HADOOP-12189 . Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible. Contributed by Zhihai Xu. (wang: rev 6736a1ab7033523ed5f304fdfed46d7f348665b4) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestCallQueueManager.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #266 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/266/)
        HADOOP-12189. Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible. Contributed by Zhihai Xu. (wang: rev 6736a1ab7033523ed5f304fdfed46d7f348665b4)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestCallQueueManager.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #266 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/266/ ) HADOOP-12189 . Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible. Contributed by Zhihai Xu. (wang: rev 6736a1ab7033523ed5f304fdfed46d7f348665b4) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestCallQueueManager.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2193 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2193/)
        HADOOP-12189. Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible. Contributed by Zhihai Xu. (wang: rev 6736a1ab7033523ed5f304fdfed46d7f348665b4)

        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestCallQueueManager.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2193 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2193/ ) HADOOP-12189 . Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible. Contributed by Zhihai Xu. (wang: rev 6736a1ab7033523ed5f304fdfed46d7f348665b4) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestCallQueueManager.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #255 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/255/)
        HADOOP-12189. Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible. Contributed by Zhihai Xu. (wang: rev 6736a1ab7033523ed5f304fdfed46d7f348665b4)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestCallQueueManager.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #255 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/255/ ) HADOOP-12189 . Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible. Contributed by Zhihai Xu. (wang: rev 6736a1ab7033523ed5f304fdfed46d7f348665b4) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestCallQueueManager.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2212 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2212/)
        HADOOP-12189. Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible. Contributed by Zhihai Xu. (wang: rev 6736a1ab7033523ed5f304fdfed46d7f348665b4)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestCallQueueManager.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2212 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2212/ ) HADOOP-12189 . Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible. Contributed by Zhihai Xu. (wang: rev 6736a1ab7033523ed5f304fdfed46d7f348665b4) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestCallQueueManager.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #263 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/263/)
        HADOOP-12189. Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible. Contributed by Zhihai Xu. (wang: rev 6736a1ab7033523ed5f304fdfed46d7f348665b4)

        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestCallQueueManager.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #263 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/263/ ) HADOOP-12189 . Improve CallQueueManager#swapQueue to make queue elements drop nearly impossible. Contributed by Zhihai Xu. (wang: rev 6736a1ab7033523ed5f304fdfed46d7f348665b4) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestCallQueueManager.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java

          People

          • Assignee:
            zxu zhihai xu
            Reporter:
            zxu zhihai xu
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development