Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-1326 New consumer checklist
  3. KAFKA-2168

New consumer poll() can block other calls like position(), commit(), and close() indefinitely

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0.0
    • Component/s: clients, consumer
    • Labels:
      None

      Description

      The new consumer is currently using very coarse-grained synchronization. For most methods this isn't a problem since they finish quickly once the lock is acquired, but poll() might run for a long time (and commonly will since polling with long timeouts is a normal use case). This means any operations invoked from another thread may block until the poll() call completes.

      Some example use cases where this can be a problem:

      • A shutdown hook is registered to trigger shutdown and invokes close(). It gets invoked from another thread and blocks indefinitely.
      • User wants to manage offset commit themselves in a background thread. If the commit policy is not purely time based, it's not currently possibly to make sure the call to commit() will be processed promptly.

      Two possible solutions to this:
      1. Make sure a lock is not held during the actual select call. Since we have multiple layers (KafkaConsumer -> NetworkClient -> Selector -> nio Selector) this is probably hard to make work cleanly since locking is currently only performed at the KafkaConsumer level and we'd want it unlocked around a single line of code in Selector.
      2. Wake up the selector before synchronizing for certain operations. This would require some additional coordination to make sure the caller of wakeup() is able to acquire the lock promptly (instead of, e.g., the poll() thread being woken up and then promptly reacquiring the lock with a subsequent long poll() call).

        Attachments

        1. KAFKA-2168_2015-06-30_10:54:22.patch
          6 kB
          Jason Gustafson
        2. KAFKA-2168_2015-06-23_09:39:07.patch
          2 kB
          Jason Gustafson
        3. KAFKA-2168_2015-06-22_16:34:37.patch
          368 kB
          Jason Gustafson
        4. KAFKA-2168_2015-06-19_09:19:02.patch
          367 kB
          Jason Gustafson
        5. KAFKA-2168_2015-06-18_14:39:36.patch
          363 kB
          Jason Gustafson
        6. KAFKA-2168_2015-06-11_14:09:59.patch
          329 kB
          Jason Gustafson
        7. KAFKA-2168_2015-06-05_12:44:48.patch
          268 kB
          Jason Gustafson
        8. KAFKA-2168_2015-06-05_12:01:28.patch
          263 kB
          Jason Gustafson
        9. KAFKA-2168_2015-06-04_14:36:04.patch
          253 kB
          Jason Gustafson
        10. KAFKA-2168_2015-06-03_21:06:42.patch
          244 kB
          Jason Gustafson
        11. KAFKA-2168_2015-06-03_18:20:23.patch
          241 kB
          Jason Gustafson
        12. KAFKA-2168_2015-06-02_17:09:37.patch
          153 kB
          Jason Gustafson
        13. KAFKA-2168_2015-06-01_16:03:38.patch
          91 kB
          Jason Gustafson
        14. KAFKA-2168.patch
          23 kB
          Jason Gustafson

          Issue Links

            Activity

              People

              • Assignee:
                hachikuji Jason Gustafson
                Reporter:
                ewencp Ewen Cheslack-Postava
                Reviewer:
                Guozhang Wang
              • Votes:
                1 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: