Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.8.2.0
    • Fix Version/s: 0.9.0.0
    • Component/s: purgatory
    • Labels:
      None

      Description

      This is a new design of purgatory based on Hierarchical Timing Wheels.

      https://cwiki.apache.org/confluence/display/KAFKA/Purgatory+Redesign+Proposal

      1. KAFKA-1989.patch
        36 kB
        Yasuhiro Matsuda
      2. KAFKA-1989_2015-04-08_14:29:51.patch
        87 kB
        Yasuhiro Matsuda
      3. KAFKA-1989_2015-04-08_13:27:59.patch
        84 kB
        Yasuhiro Matsuda
      4. KAFKA-1989_2015-04-07_14:59:33.patch
        79 kB
        Yasuhiro Matsuda
      5. KAFKA-1989_2015-04-01_13:49:58.patch
        70 kB
        Yasuhiro Matsuda
      6. KAFKA-1989_2015-03-20_08:44:57.patch
        59 kB
        Yasuhiro Matsuda

        Activity

        Hide
        onurkaraman Onur Karaman added a comment -
        Show
        onurkaraman Onur Karaman added a comment - Here's the ticket: https://issues.apache.org/jira/browse/KAFKA-2226
        Hide
        yasuhiro.matsuda Yasuhiro Matsuda added a comment -

        Thanks. I am looking into it. Would you file a separate ticket for this?

        Show
        yasuhiro.matsuda Yasuhiro Matsuda added a comment - Thanks. I am looking into it. Would you file a separate ticket for this?
        Hide
        onurkaraman Onur Karaman added a comment - - edited

        Hi Yasuhiro Matsuda.

        A NullPointerException sometimes shows up in TimerTaskList.remove while running TestPurgatoryPerformance. I’m on the HEAD of trunk.

        > ./bin/kafka-run-class.sh kafka.TestPurgatoryPerformance --key-space-size 100000 --keys 3 --num 100000 --pct50 50 --pct75 75 --rate 1000 --size 50 --timeout 20
        SLF4J: Class path contains multiple SLF4J bindings.
        SLF4J: Found binding in [jar:file:/Users/okaraman/code/kafka/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
        SLF4J: Found binding in [jar:file:/Users/okaraman/code/kafka/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
        SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
        SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
        [2015-05-27 10:02:14,782] ERROR [completion thread], Error due to  (kafka.TestPurgatoryPerformance$CompletionQueue$$anon$1)
        java.lang.NullPointerException
          at kafka.utils.timer.TimerTaskList.remove(TimerTaskList.scala:80)
          at kafka.utils.timer.TimerTaskEntry.remove(TimerTaskList.scala:128)
          at kafka.utils.timer.TimerTask$class.cancel(TimerTask.scala:27)
          at kafka.server.DelayedOperation.cancel(DelayedOperation.scala:50)
          at kafka.server.DelayedOperation.forceComplete(DelayedOperation.scala:71)
          at kafka.TestPurgatoryPerformance$CompletionQueue$$anon$1.doWork(TestPurgatoryPerformance.scala:263)
          at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
        
        Show
        onurkaraman Onur Karaman added a comment - - edited Hi Yasuhiro Matsuda . A NullPointerException sometimes shows up in TimerTaskList.remove while running TestPurgatoryPerformance. I’m on the HEAD of trunk. > ./bin/kafka-run-class.sh kafka.TestPurgatoryPerformance --key-space-size 100000 --keys 3 --num 100000 --pct50 50 --pct75 75 --rate 1000 --size 50 --timeout 20 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/Users/okaraman/code/kafka/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/Users/okaraman/code/kafka/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http: //www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] [2015-05-27 10:02:14,782] ERROR [completion thread], Error due to (kafka.TestPurgatoryPerformance$CompletionQueue$$anon$1) java.lang.NullPointerException at kafka.utils.timer.TimerTaskList.remove(TimerTaskList.scala:80) at kafka.utils.timer.TimerTaskEntry.remove(TimerTaskList.scala:128) at kafka.utils.timer.TimerTask$class.cancel(TimerTask.scala:27) at kafka.server.DelayedOperation.cancel(DelayedOperation.scala:50) at kafka.server.DelayedOperation.forceComplete(DelayedOperation.scala:71) at kafka.TestPurgatoryPerformance$CompletionQueue$$anon$1.doWork(TestPurgatoryPerformance.scala:263) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
        Hide
        guozhang Guozhang Wang added a comment -

        This is great, thanks Yasuhiro Matsuda!

        Show
        guozhang Guozhang Wang added a comment - This is great, thanks Yasuhiro Matsuda !
        Hide
        junrao Jun Rao added a comment -

        Thanks for the latest patch. +1. Committed to trunk after some minor changes in the comments.

        Show
        junrao Jun Rao added a comment - Thanks for the latest patch. +1. Committed to trunk after some minor changes in the comments.
        Hide
        yasuhiro.matsuda Yasuhiro Matsuda added a comment -

        Updated reviewboard https://reviews.apache.org/r/31568/diff/
        against branch origin/trunk

        Show
        yasuhiro.matsuda Yasuhiro Matsuda added a comment - Updated reviewboard https://reviews.apache.org/r/31568/diff/ against branch origin/trunk
        Hide
        yasuhiro.matsuda Yasuhiro Matsuda added a comment -

        Updated reviewboard https://reviews.apache.org/r/31568/diff/
        against branch origin/trunk

        Show
        yasuhiro.matsuda Yasuhiro Matsuda added a comment - Updated reviewboard https://reviews.apache.org/r/31568/diff/ against branch origin/trunk
        Hide
        yasuhiro.matsuda Yasuhiro Matsuda added a comment -

        Updated reviewboard https://reviews.apache.org/r/31568/diff/
        against branch origin/trunk

        Show
        yasuhiro.matsuda Yasuhiro Matsuda added a comment - Updated reviewboard https://reviews.apache.org/r/31568/diff/ against branch origin/trunk
        Hide
        yasuhiro.matsuda Yasuhiro Matsuda added a comment -

        Updated reviewboard https://reviews.apache.org/r/31568/diff/
        against branch origin/trunk

        Show
        yasuhiro.matsuda Yasuhiro Matsuda added a comment - Updated reviewboard https://reviews.apache.org/r/31568/diff/ against branch origin/trunk
        Hide
        yasuhiro.matsuda Yasuhiro Matsuda added a comment -

        Updated reviewboard https://reviews.apache.org/r/31568/diff/
        against branch origin/trunk

        Show
        yasuhiro.matsuda Yasuhiro Matsuda added a comment - Updated reviewboard https://reviews.apache.org/r/31568/diff/ against branch origin/trunk
        Hide
        yasuhiro.matsuda Yasuhiro Matsuda added a comment -

        Created reviewboard https://reviews.apache.org/r/31568/diff/
        against branch origin/trunk

        Show
        yasuhiro.matsuda Yasuhiro Matsuda added a comment - Created reviewboard https://reviews.apache.org/r/31568/diff/ against branch origin/trunk

          People

          • Assignee:
            yasuhiro.matsuda Yasuhiro Matsuda
            Reporter:
            yasuhiro.matsuda Yasuhiro Matsuda
          • Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development