Kafka
  1. Kafka
  2. KAFKA-1024

possible memory leak in 0.8 beta1 producer with G1GC

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Not A Problem
    • Affects Version/s: 0.8.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:
      java version "1.7.0_17"
      Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
      Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)

      Description

      I have this in my pom.xml

      <dependency>
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka_2.9.2</artifactId>
      <version>0.8.0-beta1</version>
      </dependency>
      <dependency>
      <groupId>org.scala-lang</groupId>
      <artifactId>scala-library</artifactId>
      <version>2.9.2</version>
      </dependency>
      <dependency>
      <groupId>org.xerial.snappy</groupId>
      <artifactId>snappy-java</artifactId>
      <version>1.0.4.1</version>
      </dependency>

      I'm using snappy compression codec and producing about 7k msg/sec on average. I'm producing batches up to 10 messages per batch with null keys to a topic with 16 partitions, no replication. Xms and Xmx are 64MB and I'm using XX:+UseG1GC. After about 12 hours of operation heap usage hits right up against the 64MB limit and the producer drops to about 4k msg/sec because of the GC pressure. When I restart the process the heap usage goes back to normal (around 30MB) and the producer does 7k msg/sec again.

      What else can I provide to help debug this?

        Activity

        Jason Toffaletti made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Jay Kreps made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Not A Problem [ 8 ]
        Jason Toffaletti made changes -
        Description I have this in my pom.xml

                <dependency>
                    <groupId>org.apache.kafka</groupId>
                    <artifactId>kafka_2.9.2</artifactId>
                    <version>0.8.0-beta1</version>
                </dependency>
        <dependency>
                    <groupId>org.scala-lang</groupId>
                    <artifactId>scala-library</artifactId>
                    <version>2.9.2</version>
                </dependency>
                <dependency>
                    <groupId>org.xerial.snappy</groupId>
                    <artifactId>snappy-java</artifactId>
                    <version>1.0.4.1</version>
                </dependency>

        I'm using snappy compression codec and producing about 7k msg/sec on average. I'm producing batches up to 10 messages per batch with null keys to a topic with 16 partitions, no replication. Xms and Xmx are 64MB and I'm using XX:+UseG1GC. After about 12 hours of operation heap usage hits right up against the 64MB limit and the producer drops to about 4k msg/sec because of the GC pressure. When I restart the process the heap usage goes back to normal (around 30MB) and the producer does 7k msg/sec again.

        What else can I provide to help debug this?
        I have this in my pom.xml

                <dependency>
                    <groupId>org.apache.kafka</groupId>
                    <artifactId>kafka_2.9.2</artifactId>
                    <version>0.8.0-beta1</version>
                </dependency>
                <dependency>
                    <groupId>org.scala-lang</groupId>
                    <artifactId>scala-library</artifactId>
                    <version>2.9.2</version>
                </dependency>
                <dependency>
                    <groupId>org.xerial.snappy</groupId>
                    <artifactId>snappy-java</artifactId>
                    <version>1.0.4.1</version>
                </dependency>

        I'm using snappy compression codec and producing about 7k msg/sec on average. I'm producing batches up to 10 messages per batch with null keys to a topic with 16 partitions, no replication. Xms and Xmx are 64MB and I'm using XX:+UseG1GC. After about 12 hours of operation heap usage hits right up against the 64MB limit and the producer drops to about 4k msg/sec because of the GC pressure. When I restart the process the heap usage goes back to normal (around 30MB) and the producer does 7k msg/sec again.

        What else can I provide to help debug this?
        Jason Toffaletti made changes -
        Description I have this in my pom.xml

                <dependency>
                    <groupId>org.apache.kafka</groupId>
                    <artifactId>kafka_2.9.2</artifactId>
                    <version>0.8.0-beta1</version>
                </dependency>

        I'm using snappy compression codec and producing about 7k msg/sec on average. I'm producing batches up to 10 messages per batch with null keys to a topic with 16 partitions, no replication. Xms and Xmx are 64MB and I'm using XX:+UseG1GC. After about 12 hours of operation heap usage hits right up against the 64MB limit and the producer drops to about 4k msg/sec because of the GC pressure. When I restart the process the heap usage goes back to normal (around 30MB) and the producer does 7k msg/sec again.

        What else can I provide to help debug this?
        I have this in my pom.xml

                <dependency>
                    <groupId>org.apache.kafka</groupId>
                    <artifactId>kafka_2.9.2</artifactId>
                    <version>0.8.0-beta1</version>
                </dependency>
        <dependency>
                    <groupId>org.scala-lang</groupId>
                    <artifactId>scala-library</artifactId>
                    <version>2.9.2</version>
                </dependency>
                <dependency>
                    <groupId>org.xerial.snappy</groupId>
                    <artifactId>snappy-java</artifactId>
                    <version>1.0.4.1</version>
                </dependency>

        I'm using snappy compression codec and producing about 7k msg/sec on average. I'm producing batches up to 10 messages per batch with null keys to a topic with 16 partitions, no replication. Xms and Xmx are 64MB and I'm using XX:+UseG1GC. After about 12 hours of operation heap usage hits right up against the 64MB limit and the producer drops to about 4k msg/sec because of the GC pressure. When I restart the process the heap usage goes back to normal (around 30MB) and the producer does 7k msg/sec again.

        What else can I provide to help debug this?
        Jason Toffaletti made changes -
        Field Original Value New Value
        Description I have this in my pom.xml

        {noformat}
                <dependency>
                    <groupId>org.apache.kafka</groupId>
                    <artifactId>kafka_2.9.2</artifactId>
                    <version>0.8.0-beta1</version>
                </dependency>
        {noformat}

        I'm using snappy compression codec and producing about 7k msg/sec on average. I'm producing batches up to 10 messages per batch with null keys to a topic with 16 partitions, no replication. Xms and Xmx are 64MB and I'm using XX:+UseG1GC. After about 12 hours of operation heap usage hits right up against the 64MB limit and the producer drops to about 4k msg/sec because of the GC pressure. When I restart the process the heap usage goes back to normal (around 30MB) and the producer does 7k msg/sec again.

        What else can I provide to help debug this?
        I have this in my pom.xml

                <dependency>
                    <groupId>org.apache.kafka</groupId>
                    <artifactId>kafka_2.9.2</artifactId>
                    <version>0.8.0-beta1</version>
                </dependency>

        I'm using snappy compression codec and producing about 7k msg/sec on average. I'm producing batches up to 10 messages per batch with null keys to a topic with 16 partitions, no replication. Xms and Xmx are 64MB and I'm using XX:+UseG1GC. After about 12 hours of operation heap usage hits right up against the 64MB limit and the producer drops to about 4k msg/sec because of the GC pressure. When I restart the process the heap usage goes back to normal (around 30MB) and the producer does 7k msg/sec again.

        What else can I provide to help debug this?
        Jason Toffaletti created issue -

          People

          • Assignee:
            Unassigned
            Reporter:
            Jason Toffaletti
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development