Kafka
  1. Kafka
  2. KAFKA-117

The FetcherRunnable busy waits on empty fetch requests

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.7
    • Fix Version/s: 0.7
    • Component/s: None
    • Labels:
      None

      Description

      The FetcherRunnable busy waits on empty fetch requests by skipping the backoff logic. This can fill up the disk space due to the public access log being filled up. Also the CPU usage shoots up to 100%.

      1. KAFKA-117.patch
        0.6 kB
        Neha Narkhede
      2. KAFKA-117-v2.patch
        3 kB
        Neha Narkhede
      3. KAFKA-117-v3.patch
        4 kB
        Neha Narkhede

        Activity

        Neha Narkhede created issue -
        Neha Narkhede made changes -
        Field Original Value New Value
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Neha Narkhede added a comment -

        The bug was in the shallowValidBytes logic where it returns a negative value when dealing with an empty fetch request.

        Corrected the logic of shallowValidBytes to return 0 when there is no data in the ByteBufferMessageSet created from an empty fetch request.

        Show
        Neha Narkhede added a comment - The bug was in the shallowValidBytes logic where it returns a negative value when dealing with an empty fetch request. Corrected the logic of shallowValidBytes to return 0 when there is no data in the ByteBufferMessageSet created from an empty fetch request.
        Neha Narkhede made changes -
        Attachment KAFKA-117.patch [ 12491393 ]
        Hide
        Jun Rao added a comment -

        We probably should just have a method validBytes that returns the valid bytes in the original message (ie, if the message is compressed, the bytes refer to the compressed bytes). There is no real need for knowing the total valid bytes on a compressed message set.

        Show
        Jun Rao added a comment - We probably should just have a method validBytes that returns the valid bytes in the original message (ie, if the message is compressed, the bytes refer to the compressed bytes). There is no real need for knowing the total valid bytes on a compressed message set.
        Hide
        Neha Narkhede added a comment -

        I think your suggestion makes sense. The deepValidBytes method is useless and never used, even by the unit tests. It is better to have just a single API for calculating valid bytes.

        Show
        Neha Narkhede added a comment - I think your suggestion makes sense. The deepValidBytes method is useless and never used, even by the unit tests. It is better to have just a single API for calculating valid bytes.
        Hide
        Neha Narkhede added a comment -

        Deleted the deepValidBytes API. Made the shallowValidBytes API private and had the validBytes API point to shallowValidBytes.

        validBytes() API will always give you the number of compressed bytes in the ByteBufferMessageSet, if it is compressed.

        Show
        Neha Narkhede added a comment - Deleted the deepValidBytes API. Made the shallowValidBytes API private and had the validBytes API point to shallowValidBytes. validBytes() API will always give you the number of compressed bytes in the ByteBufferMessageSet, if it is compressed.
        Neha Narkhede made changes -
        Attachment KAFKA-117-v2.patch [ 12491421 ]
        Hide
        Neha Narkhede added a comment -

        Includes all changes from v2 patch + a unit test to cover the valid bytes for empty message sets

        Show
        Neha Narkhede added a comment - Includes all changes from v2 patch + a unit test to cover the valid bytes for empty message sets
        Neha Narkhede made changes -
        Attachment KAFKA-117-v3.patch [ 12491427 ]
        Hide
        Jun Rao added a comment -

        +1 on v3.

        Show
        Jun Rao added a comment - +1 on v3.
        Jun Rao made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Neha Narkhede
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development