Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-1430

Purgatory redesign

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 0.8.2.0
    • 0.8.2.0
    • core
    • None

    Description

      We have seen 2 main issues with the Purgatory.

      1. There is no atomic checkAndWatch functionality. So, a client typically first checks whether a request is satisfied or not and then register the watcher. However, by the time the watcher is registered, the registered item could already be satisfied. This item won't be satisfied until the next update happens or the delayed time expires, which means the watched item could be delayed.

      2. FetchRequestPurgatory doesn't quite work. This is because the current design tries to incrementally maintain the accumulated bytes ready for fetch. However, this is difficult since the right time to check whether a fetch (for regular consumer) request is satisfied is when the high watermark moves. At that point, it's hard to figure out how many bytes we should incrementally add to each pending fetch request.

      The problem has been reported in KAFKA-1150 and KAFKA-703.

      Attachments

        1. KAFKA-1430_2014-06-05_17:07:37.patch
          108 kB
          Guozhang Wang
        2. KAFKA-1430_2014-06-05_17:13:13.patch
          129 kB
          Guozhang Wang
        3. KAFKA-1430_2014-06-05_17:40:53.patch
          128 kB
          Guozhang Wang
        4. KAFKA-1430_2014-06-10_11:22:06.patch
          131 kB
          Guozhang Wang
        5. KAFKA-1430_2014-06-10_11:26:02.patch
          132 kB
          Guozhang Wang
        6. KAFKA-1430_2014-07-11_10:59:13.patch
          132 kB
          Guozhang Wang
        7. KAFKA-1430_2014-07-21_12:53:39.patch
          141 kB
          Guozhang Wang
        8. KAFKA-1430_2014-07-25_09:52:43.patch
          138 kB
          Guozhang Wang
        9. KAFKA-1430_2014-07-28_11:30:23.patch
          164 kB
          Guozhang Wang
        10. KAFKA-1430_2014-07-31_15:04:33.patch
          167 kB
          Guozhang Wang
        11. KAFKA-1430_2014-08-05_14:54:21.patch
          158 kB
          Guozhang Wang
        12. KAFKA-1430.patch
          146 kB
          Guozhang Wang
        13. KAFKA-1430.patch
          141 kB
          Guozhang Wang
        14. KAFKA-1430.patch
          138 kB
          Guozhang Wang
        15. KAFKA-1430.patch
          134 kB
          Guozhang Wang
        16. KAFKA-1430.patch
          50 kB
          Guozhang Wang

        Issue Links

          Activity

            People

              guozhang Guozhang Wang
              junrao Jun Rao
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: