Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-1417

Very slow initial high-level consumer startup in low traffic/blocking fetch scenario

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.1
    • Component/s: consumer
    • Labels:
      None

      Description

      We're seeing very slow startup times when starting a high level consumer in a low traffic/blocking fetch type setup. The example we've come across has a consumer that is set up to use 3 topics and uses a 20s/1 byte fetch timeout. What happens is that the leader finder thread adds partitions one by one and since the offset is not know this causes a call to figure out the offset. This call uses the fetcher threads simple consumer instance and locks around the call. Initially this is not a problem, but as soon as the fetcher thread has some partitions it will start fetching and since this is a low traffic situation the fetch will at least sometimes take up to 20s (again locking around the simple consumer). This leads to behavior like:

      1. Finder thread adds a partition
      2. Data thread notices it has partitions to fetch data for, locks the consumer for 20s
      3. Finder thread tries to add a partition, tries to lock consumer and blocks for 20s
      4. Rinse, repeat for each partition

        Attachments

          Activity

            People

            • Assignee:
              nehanarkhede Neha Narkhede
              Reporter:
              smeder Sam Meder
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: