Kafka
  1. Kafka
  2. KAFKA-703

A fetch request in Fetch Purgatory can double count the bytes from the same delayed produce request

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.8.1
    • Fix Version/s: 0.8.2.0
    • Component/s: purgatory
    • Labels:
      None

      Description

      When a producer request is handled, the fetch purgatory is checked to ensure any fetch requests are satisfied. When the produce request is satisfied we do the check again and if the same fetch request was still in the fetch purgatory it would end up double counting the bytes received.

      Possible Solutions

      1. In the delayed produce request case, do the check only after the produce request is satisfied. This could potentially delay the fetch request from being satisfied.
      2. Remove dependency of fetch request on produce request and just look at the last logical log offset (which should mostly be cached). This would need the replica.fetch.min.bytes to be number of messages rather than bytes. This also helps KAFKA-671 in that we would no longer need to pass the ProduceRequest object to the producer purgatory and hence not have to consume any memory.

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          598d 49m 1 Guozhang Wang 04/Sep/14 22:56
          Tony Stevenson made changes -
          Workflow Apache Kafka Workflow [ 13051319 ] no-reopen-closed, patch-avail [ 13053670 ]
          Tony Stevenson made changes -
          Workflow no-reopen-closed, patch-avail [ 12745914 ] Apache Kafka Workflow [ 13051319 ]
          Guozhang Wang made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Guozhang Wang added a comment -

          This problem is resolved in the purgatory / API redesign: KAFKA-1583. Closing now.

          Show
          Guozhang Wang added a comment - This problem is resolved in the purgatory / API redesign: KAFKA-1583 . Closing now.
          Guozhang Wang made changes -
          Link This issue Is contained by KAFKA-1430 [ KAFKA-1430 ]
          Hide
          Neha Narkhede added a comment -

          Moving to 0.8.2

          Show
          Neha Narkhede added a comment - Moving to 0.8.2
          Neha Narkhede made changes -
          Fix Version/s 0.8.2 [ 12326167 ]
          Fix Version/s 0.8.1 [ 12322960 ]
          Neha Narkhede made changes -
          Labels p3
          Neha Narkhede made changes -
          Affects Version/s 0.8.1 [ 12322960 ]
          Affects Version/s 0.8 [ 12317244 ]
          Neha Narkhede made changes -
          Labels p3
          Jay Kreps made changes -
          Fix Version/s 0.8.1 [ 12322960 ]
          Fix Version/s 0.8 [ 12317244 ]
          Hide
          Sriram Subramanian added a comment -

          Can we move this jira to the next version since we have decided to punt this?

          Show
          Sriram Subramanian added a comment - Can we move this jira to the next version since we have decided to punt this?
          Neha Narkhede made changes -
          Assignee Sriram Subramanian [ sriramsub ]
          Jun Rao made changes -
          Priority Major [ 3 ] Blocker [ 1 ]
          Sriram Subramanian made changes -
          Field Original Value New Value
          Assignee Joel Koshy [ jjkoshy ]
          Sriram Subramanian created issue -

            People

            • Assignee:
              Sriram Subramanian
              Reporter:
              Sriram Subramanian
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development