Kafka
  1. Kafka
  2. KAFKA-1154

replicas may not have consistent data after becoming follower

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.1
    • Fix Version/s: 0.8.1
    • Component/s: core
    • Labels:
      None

      Description

      This is an issued introduced in KAFKA-1001. The issue is that in ReplicaManager.makeFollowers(), we truncate the log before marking the replica as the follower. New messages from the producer can still be added to the log after the log is truncated, but before the replica is marked as the follower. Those newly produced messages can actually be committed, which implies those truncated messages are also committed. However, the new leader is not guaranteed to have those truncated messages.

        Activity

        Hide
        Jun Rao added a comment -

        The fix is to switch the ordering in ReplicaManager.makeFollowers(). We should first mark the replica as the follower before truncating the log. This will ensure that after the log is truncated, no new messages from the produce requests will be added to the log.

        Show
        Jun Rao added a comment - The fix is to switch the ordering in ReplicaManager.makeFollowers(). We should first mark the replica as the follower before truncating the log. This will ensure that after the log is truncated, no new messages from the produce requests will be added to the log.
        Hide
        Jun Rao added a comment -

        Created reviewboard https://reviews.apache.org/r/15938/
        against branch origin/trunk

        Show
        Jun Rao added a comment - Created reviewboard https://reviews.apache.org/r/15938/ against branch origin/trunk
        Hide
        Jun Rao added a comment -

        Attach a patch. Also made a minor change in KafkaApis to only expose data up to the high watermark to the debugger consumer. Data after the high watermark can be truncated and therefore could change after the consumer has read it. This fix will prevent the ReplicaVerificationTool to read stale data.

        Show
        Jun Rao added a comment - Attach a patch. Also made a minor change in KafkaApis to only expose data up to the high watermark to the debugger consumer. Data after the high watermark can be truncated and therefore could change after the consumer has read it. This fix will prevent the ReplicaVerificationTool to read stale data.
        Hide
        Neha Narkhede added a comment -

        We used to do this, looks like this was a regression introduced in KAFKA-1001.

        1. ReplicaManager
        Do we still need this "TODO: the above may need to be fixed later" ?

        2. We had added the ability for a special consumer to read the replica log for troubleshooting. This patch takes that convenience away. We should probably look for another way to prevent the replica verification tool from giving false negatives. Can it use a different consumer id?

        Show
        Neha Narkhede added a comment - We used to do this, looks like this was a regression introduced in KAFKA-1001 . 1. ReplicaManager Do we still need this "TODO: the above may need to be fixed later" ? 2. We had added the ability for a special consumer to read the replica log for troubleshooting. This patch takes that convenience away. We should probably look for another way to prevent the replica verification tool from giving false negatives. Can it use a different consumer id?
        Hide
        Jun Rao added a comment -

        Thanks for the review. Committed to trunk after addressing the minor review comments.

        Show
        Jun Rao added a comment - Thanks for the review. Committed to trunk after addressing the minor review comments.

          People

          • Assignee:
            Jun Rao
            Reporter:
            Jun Rao
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development