Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-1469

Unable to deploy large topologies on apache storm

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Reopened
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.0.0, 2.0.0
    • Fix Version/s: None
    • Component/s: storm-core

      Description

      When deploying to a nimbus a topology which is larger in size >17MB, we get an exception. In storm 0.9.3 this could be mitigated by using the following config on the storm.yaml to increse the buffer size to handle the topology size. i.e. 50MB would be

      nimbus.thrift.max_buffer_size: 50000000

      This configuration does not resolve the issue in the master branch of storm and we cannot deploy topologies which are large in size.

      Here is the log on the client side when attempting to deploy to the nimbus node:
      java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException
      at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:251) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:272) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:155) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      at com.trustwave.siem.storm.topology.deployer.TopologyDeployer.deploy(TopologyDeployer.java:149) [siem-ng-storm-deployer-cloud.jar:]
      at com.trustwave.siem.storm.topology.deployer.TopologyDeployer.main(TopologyDeployer.java:87) [siem-ng-storm-deployer-cloud.jar:]
      Caused by: org.apache.thrift7.transport.TTransportException
      at org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      at org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      at org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      at org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      at org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      at org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:77) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      at backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:238) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      at backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:222) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:237) ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
      ... 4 more

      Here is the log on the server side (nimbus.log):

      2016-01-13 10:48:07.206 o.a.s.d.nimbus [INFO] Cleaning inbox ... deleted: stormjar-c8666220-fa19-426b-a7e4-c62dfb57f1f0.jar
      2016-01-13 10:55:09.823 o.a.s.d.nimbus [INFO] Uploading file from client to /var/storm-data/nimbus/inbox/stormjar-80ecdf05-6a25-4281-8c78-10062ac5e396.jar
      2016-01-13 10:55:11.910 o.a.s.d.nimbus [INFO] Finished uploading file from client: /var/storm-data/nimbus/inbox/stormjar-80ecdf05-6a25-4281-8c78-10062ac5e396.jar
      2016-01-13 10:55:12.084 o.a.t.s.AbstractNonblockingServer$FrameBuffer [WARN] Exception while invoking!
      org.apache.thrift7.transport.TTransportException: Frame size (17435758) larger than max length (16384000)!
      at org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:137)
      at org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101)
      at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86)
      at org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
      at org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
      at org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
      at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:27)
      at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158)
      at org.apache.thrift7.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518)
      at org.apache.thrift7.server.Invocation.run(Invocation.java:18)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kishorvpatil Kishor Patil
                Reporter:
                rudra.sharma Rudra Sharma
              • Votes:
                2 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m