[FLUME-2812] Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.Error: Maximum permit count exceeded

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 1.8.0
• Component/s: Channel, Sinks+Sources

Environment

*OS INFO*
CentOS release 6.6 (Final)
Kernel \r on an \m

*JAVA INFO*
java version "1.8.0_40"
Java(TM) SE Runtime Environment (build 1.8.0_40-b26)
Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)

• Assignee: Denes Arvay
• Reporter: Rollin Crittendon
• Created: 14/Oct/15 20:23
• Updated: 05/Nov/16 07:09
• Resolved: 05/Nov/16 05:24

Description

We are finding that around after an hour or so of heavy processing of Flume data in an agent we are getting the following exception. This is after processing about 5-7 k lines/second during that time.

The configuration of this agent is using a Kafka source, the one that comes with 1.6.0.

It is also using a Memory channel, and a Thrift sink.

=======
Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.Error: Maximum permit count exceeded
at java.util.concurrent.Semaphore$Sync.tryReleaseShared(Semaphore.java:192)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
at java.util.concurrent.Semaphore.release(Semaphore.java:609)
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:147)
at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:379)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
=======

The above error is from standard error when running the Flume agent. The effect is that the "SinkRunner-PollingRunner-DefaultSinkProcessor" thread disappears from the agent, this can be seen on a JMX console.
For us, this means that the Flume agent needs to get restarted. It is an error that is terminal in that instance of the Java process due to the thread disappearing as a result.

It sounds like something in JDK 7+ got stricter?!

Issue Links

• links to: GitHub Pull Request #83