Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: v1.2.0, v1.3.0, v1.4.0
    • Fix Version/s: None
    • Component/s: None
    • Release Note:
      HDFS Sink memory usage best practices.

      Description

      We run a 3 node/1 collector test cluster pushing about 350events/sec per node... Not really high stress, but just something to evaluate flume with.

      Consistently our collector has been dying because of an OOMError killing the SinkRunner after running for about 30-40 hours(seems pretty consistent as we've had it 3 times now).

      Suspected cause would be a memory leak somewhere in HdfsSink. The feeder nodes which run AvroSink instead of HdfsSink have been up and running for about a week without restarts.

      flume-load/act-wap02/2012-06-26-17.1340697637324.tmp, packetSize=65557, chunksPerPacket=127, bytesCurBlock=29731328
      java.lang.OutOfMemoryError: GC overhead limit exceeded
      2012-06-26 17:12:56,080 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:411)] process failed
      java.lang.OutOfMemoryError: GC overhead limit exceeded
      at java.util.Arrays.copyOfRange(Arrays.java:3209)
      at java.lang.String.<init>(String.java:215)
      at java.lang.StringBuilder.toString(StringBuilder.java:430)
      at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:306)
      at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:367)
      at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
      at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
      at java.lang.Thread.run(Thread.java:619)
      Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.OutOfMemoryError: GC overhead limit exceeded
      at java.util.Arrays.copyOfRange(Arrays.java:3209)
      at java.lang.String.<init>(String.java:215)
      at java.lang.StringBuilder.toString(StringBuilder.java:430)
      at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:306)
      at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:367)
      at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
      at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
      at java.lang.Thread.run(Thread.java:619)

      1. FLUME-1326.patch
        1 kB
        dave sinclair

        Activity

          People

          • Assignee:
            Unassigned
            Reporter:
            Juhani Connolly
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:

              Development