[STORM-1469] Unable to deploy large topologies on apache storm

• Type: Bug
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.0.0, 2.0.0
• Component/s: storm-core
• Labels: pull-request-available
• Assignee: Kishor Patil
• Reporter: Rudra Sharma
• Created: 13/Jan/16 16:02
• Updated: 28/Oct/19 15:02

Description

When deploying to a nimbus a topology which is larger in size >17MB, we get an exception. In storm 0.9.3 this could be mitigated by using the following config on the storm.yaml to increse the buffer size to handle the topology size. i.e. 50MB would be

nimbus.thrift.max_buffer_size: 50000000

This configuration does not resolve the issue in the master branch of storm and we cannot deploy topologies which are large in size.

Here is the log on the client side when attempting to deploy to the nimbus node:
java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:251) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:272) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:155) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at com.trustwave.siem.storm.topology.deployer.TopologyDeployer.deploy(TopologyDeployer.java:149) [siem-ng-storm-deployer-cloud.jar:]
at com.trustwave.siem.storm.topology.deployer.TopologyDeployer.main(TopologyDeployer.java:87) [siem-ng-storm-deployer-cloud.jar:]
Caused by: org.apache.thrift7.transport.TTransportException
at org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:77) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:238) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:222) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:237) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
... 4 more

Here is the log on the server side (nimbus.log):

2016-01-13 10:48:07.206 o.a.s.d.nimbus [INFO] Cleaning inbox ... deleted: stormjar-c8666220-fa19-426b-a7e4-c62dfb57f1f0.jar
2016-01-13 10:55:09.823 o.a.s.d.nimbus [INFO] Uploading file from client to /var/storm-data/nimbus/inbox/stormjar-80ecdf05-6a25-4281-8c78-10062ac5e396.jar
2016-01-13 10:55:11.910 o.a.s.d.nimbus [INFO] Finished uploading file from client: /var/storm-data/nimbus/inbox/stormjar-80ecdf05-6a25-4281-8c78-10062ac5e396.jar
2016-01-13 10:55:12.084 o.a.t.s.AbstractNonblockingServer$FrameBuffer [WARN] Exception while invoking!
org.apache.thrift7.transport.TTransportException: Frame size (17435758) larger than max length (16384000)!
at org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:137)
at org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101)
at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:27)
at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158)
at org.apache.thrift7.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518)
at org.apache.thrift7.server.Invocation.run(Invocation.java:18)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Issue Links

• depends upon: STORM-1661 Introduce a config to turn off blobstore acl validation in insecure mode
• links to: GitHub Pull Request #1178