Kafka
  1. Kafka
  2. KAFKA-1548

Refactor the "replica_id" in requests

    Details

    • Type: Bug Bug
    • Status: In Progress
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 0.10.2.0
    • Component/s: None
    • Labels:

      Description

      Today in many requests like fetch and offset we have a integer replica_id field, if the request is from a follower consumer it is the broker id from that follower replica, if it is from a regular consumer it could be one of the two values: "-1" for ordinary consumer, or "-2" for debugging consumer.

      Hence this replica_id field is used in two folds:

      1) Logging for trouble shooting in request logs, which can be helpful only when this is from a follower replica,

      2) Deciding if it is from the consumer or a replica to logically handle the request in different ways. For this purpose we do not really care about the actually id value.

      We probably would like to do the following improvements:

      1) Rename "replica_id" to sth. less confusing?

      2) Change the request.toString() function based on the replica_id, whether it is a positive integer (meaning from a broker replica fetcher) or -1/-2 (meaning from a regular consumer).

        Issue Links

          Activity

          Hide
          Gwen Shapira added a comment -

          I think the name replica_id is fine.

          For #2, I'm assuming the idea is to change the request printout in the log from:
          {replica_id=-1,max_wait_time=100,min_bytes=100000,topics=[{topic=test1,partitions=[

          {partition=0,fetch_offset=100,max_bytes=1000000}]},{topic=test2,partitions=[{partition=0,fetch_offset=200,max_bytes=1000000}]}]}

          to:
          {ordinary consumer,max_wait_time=100,min_bytes=100000,topics=[{topic=test1,partitions=[{partition=0,fetch_offset=100,max_bytes=1000000}

          ]},{topic=test2,partitions=[

          {partition=0,fetch_offset=200,max_bytes=1000000}

          ]}]}

          In cases where the replica_id is not from a follower.

          Did I get that right?

          Show
          Gwen Shapira added a comment - I think the name replica_id is fine. For #2, I'm assuming the idea is to change the request printout in the log from: {replica_id=-1,max_wait_time=100,min_bytes=100000,topics=[{topic=test1,partitions=[ {partition=0,fetch_offset=100,max_bytes=1000000}]},{topic=test2,partitions= [{partition=0,fetch_offset=200,max_bytes=1000000}] }]} to: {ordinary consumer,max_wait_time=100,min_bytes=100000,topics=[{topic=test1,partitions=[{partition=0,fetch_offset=100,max_bytes=1000000} ]},{topic=test2,partitions=[ {partition=0,fetch_offset=200,max_bytes=1000000} ]}]} In cases where the replica_id is not from a follower. Did I get that right?
          Hide
          Guozhang Wang added a comment -

          Hi Gwen,

          I kind of agree that for now maybe we do not need to change the name, with that then what we should probably do is:

          1. Add comments in RequestOrResponse explaining the semantics of replica id.
          2. Change the toString of Request with "Follower Replica Id", "Normal Consumer" and "Tooling Consumer" (this is for -2, the debugging consumer)

          Show
          Guozhang Wang added a comment - Hi Gwen, I kind of agree that for now maybe we do not need to change the name, with that then what we should probably do is: 1. Add comments in RequestOrResponse explaining the semantics of replica id. 2. Change the toString of Request with "Follower Replica Id", "Normal Consumer" and "Tooling Consumer" (this is for -2, the debugging consumer)
          Hide
          Santosh Pingale added a comment -

          Gwen Shapira/Guozhang Wang

          Can I pick this up?

          Show
          Santosh Pingale added a comment - Gwen Shapira / Guozhang Wang Can I pick this up?
          Hide
          ASF GitHub Bot added a comment -

          GitHub user baluchicken opened a pull request:

          https://github.com/apache/kafka/pull/2137

          KAFKA-1548 Refactor the "replica_id" in requests

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/baluchicken/kafka-1 KAFKA-1548

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/kafka/pull/2137.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #2137


          commit 4e55654c288fa18ad294796b5d58c5bc8e198ccc
          Author: Balint Molnar <balintmolnar91@gmail.com>
          Date: 2016-11-15T13:27:09Z

          KAFKA-1548 Refactor the "replica_id" in requests


          Show
          ASF GitHub Bot added a comment - GitHub user baluchicken opened a pull request: https://github.com/apache/kafka/pull/2137 KAFKA-1548 Refactor the "replica_id" in requests You can merge this pull request into a Git repository by running: $ git pull https://github.com/baluchicken/kafka-1 KAFKA-1548 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2137.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2137 commit 4e55654c288fa18ad294796b5d58c5bc8e198ccc Author: Balint Molnar <balintmolnar91@gmail.com> Date: 2016-11-15T13:27:09Z KAFKA-1548 Refactor the "replica_id" in requests

            People

            • Assignee:
              Balint Molnar
              Reporter:
              Guozhang Wang
              Reviewer:
              Neha Narkhede
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:

                Development