[STORM-874] Netty Threads do not handle Errors properly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.2-incubating, 0.10.0
• Fix Version/s: 1.0.0
• Component/s: storm-core
• Assignee: Kishor Patil
• Reporter: Kishor Patil
• Created: 18/Jun/15 05:39
• Updated: 30/Nov/15 21:00
• Resolved: 30/Nov/15 21:00

Description

When low on memory, netty thread could get OOM which if not handled correctly can lead to unexpected behavior such as netty connection leaks.

java.lang.OutOfMemoryError: Direct buffer memory
	at java.nio.Bits.reserveMemory(Bits.java:658) ~[?:1.8.0_25]
	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[?:1.8.0_25]
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[?:1.8.0_25]
	at org.jboss.netty.buffer.ChannelBuffers.directBuffer(ChannelBuffers.java:167) ~[netty-3.9.4.Final.jar:?]
	at org.jboss.netty.buffer.ChannelBuffers.directBuffer(ChannelBuffers.java:151) ~[netty-3.9.4.Final.jar:?]
	at backtype.storm.messaging.netty.MessageBatch.buffer(MessageBatch.java:101) ~[storm-core-0.9.2-incubating-security.jar:0.9.2-incubating-security]
	at backtype.storm.messaging.netty.MessageEncoder.encode(MessageEncoder.java:32) ~[storm-core-0.9.2-incubating-security.jar:0.9.2-incubating-security]
	at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:66) ~[netty-3.9.4.Final.jar:?]
	at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59) ~[netty-3.9.4.Final.jar:?]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) ~[netty-3.9.4.Final.jar:?]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) ~[netty-3.9.4.Final.jar:?]
	at org.jboss.netty.channel.Channels.write(Channels.java:704) ~[netty-3.9.4.Final.jar:?]
	at org.jboss.netty.channel.Channels.write(Channels.java:671) ~[netty-3.9.4.Final.jar:?]
	at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248) ~[netty-3.9.4.Final.jar:?]
	at backtype.storm.messaging.netty.Client.tryDeliverMessages(Client.java:226) ~[storm-core-0.9.2-incubating-security.jar:0.9.2-incubating-security]
	at backtype.storm.messaging.netty.Client.send(Client.java:173) ~[storm-core-0.9.2-incubating-security.jar:0.9.2-incubating-security]

Issue Links

• mentioned in: Page Loading...