Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-5584

Incorrect log size for topics larger than 2 GB

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 0.11.0.0
    • 0.11.0.1
    • log

    Description

      The size of a Log is calculated incorrectly due to an Integer overflow. For large topics (larger than 2 GB) this value overflows.

      This is easily observable in the reported metrics values of the path log.Log.partition.*.topic.<topicname>.Size (see attached screenshot).

      Moreover I think this breaks the size-based retention (via log.retention.bytes and retention.bytes) of large topics as well.

      I am not sure on the recommended workflow, should I open a pull request on github with a fix?

      Attachments

        1. Screen Shot 2017-07-12 at 09.10.53.png
          42 kB
          Gregor Uhlenheuer

        Issue Links

          Activity

            jeffwidman Jeff Widman added a comment -

            What version of Kafka did you observe this on?

            jeffwidman Jeff Widman added a comment - What version of Kafka did you observe this on?

            Sorry, I missed that information. I am using kafka 0.11.0.0.

            kongo2002 Gregor Uhlenheuer added a comment - Sorry, I missed that information. I am using kafka 0.11.0.0 .
            githubbot ASF GitHub Bot added a comment -

            GitHub user kongo2002 opened a pull request:

            https://github.com/apache/kafka/pull/3521

            KAFKA-5584: fix integer overflow in Log.size

            As described in KAFKA-5584(https://issues.apache.org/jira/browse/KAFKA-5584) the integer overflow in `Log.size` may lead to wrong metrics and broken size-based retention via `log.retention.bytes` or `retention.bytes` on rather large topic partitions.

            You can merge this pull request into a Git repository by running:

            $ git pull https://github.com/kongo2002/kafka KAFKA-5584

            Alternatively you can review and apply these changes as the patch at:

            https://github.com/apache/kafka/pull/3521.patch

            To close this pull request, make a commit to your master/trunk branch
            with (at least) the following in the commit message:

            This closes #3521


            commit 4ba20618a13dfd6c16b8228b319a21ae1915e857
            Author: Gregor Uhlenheuer <kongo2002@googlemail.com>
            Date: 2017-07-12T10:07:49Z

            KAFKA-5584: fix integer overflow in Log.size


            githubbot ASF GitHub Bot added a comment - GitHub user kongo2002 opened a pull request: https://github.com/apache/kafka/pull/3521 KAFKA-5584 : fix integer overflow in Log.size As described in KAFKA-5584 ( https://issues.apache.org/jira/browse/KAFKA-5584 ) the integer overflow in `Log.size` may lead to wrong metrics and broken size-based retention via `log.retention.bytes` or `retention.bytes` on rather large topic partitions. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kongo2002/kafka KAFKA-5584 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3521.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3521 commit 4ba20618a13dfd6c16b8228b319a21ae1915e857 Author: Gregor Uhlenheuer <kongo2002@googlemail.com> Date: 2017-07-12T10:07:49Z KAFKA-5584 : fix integer overflow in Log.size
            githubbot ASF GitHub Bot added a comment -

            Github user asfgit closed the pull request at:

            https://github.com/apache/kafka/pull/3521

            githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3521

            Was this a regression in 0.11.0.0 specifically?

            mihbor Michal Borowiecki added a comment - Was this a regression in 0.11.0.0 specifically?

            It seems the bug was introduced with commit e71dce89c0da50f3eccc47d0fc050c92d5a99b88 I think.

            kongo2002 Gregor Uhlenheuer added a comment - It seems the bug was introduced with commit e71dce89c0da50f3eccc47d0fc050c92d5a99b88 I think.
            ijuma Ismael Juma added a comment -

            Thanks for tracking it down kongo2002, that looks right to me.

            ijuma Ismael Juma added a comment - Thanks for tracking it down kongo2002 , that looks right to me.

            People

              Unassigned Unassigned
              kongo2002 Gregor Uhlenheuer
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: