Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-1001

Handle follower transition in batch

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.1
    • Component/s: None
    • Labels:
      None

      Description

      In KAFKA-615 we made changes to avoid fsync'ing the active segment of the log due to log roll and maintaining recovery semantics.

      One downside of the fix for that issue was that it required checkpointing the recovery point for the log many times, one for each partition that transitioned to follower state.

      In this ticket I aim to fix that issue by making the following changes:
      1. Add a new API LogManager.truncateTo(m: Map[TopicAndPartition, Long]). This method will first checkpoint the recovery point, then truncate each of the given logs to the given offset. This method will have to ensure these two things happen atomically.
      2. Change ReplicaManager to first stop fetching for all partitions changing to follower state, then call LogManager.truncateTo then complete the existing logic.

      We think this will, over all, be a good thing. The reason is that the fetching thread current does something like (a) acquire lock, (b) fetch partitions, (c) write data to logs, (d) release locks. Since we currently remove fetchers one at a time this requires acquiring the fetcher lock, and hence generally blocking for half of the read/write cycle for each partition. By doing this in bulk we will avoid reacquiring the lock over and over for each change.

        Attachments

        1. KAFKA-1001.patch
          35 kB
          Guozhang Wang
        2. KAFKA-1001_2013-10-29_11:27:20.patch
          41 kB
          Guozhang Wang
        3. KAFKA-1001_2013-10-28_15:13:42.patch
          41 kB
          Guozhang Wang
        4. KAFKA-1001_2013-10-28_11:19:47.patch
          41 kB
          Guozhang Wang
        5. KAFKA-1001_2013-10-25_11:27:24.patch
          41 kB
          Guozhang Wang
        6. KAFKA-1001_2013-10-21_13:35:41.patch
          39 kB
          Guozhang Wang

          Activity

            People

            • Assignee:
              guozhang Guozhang Wang
              Reporter:
              jkreps Jay Kreps
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: