Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-11727

Streaming error while Bootstraping Materialized View

Agile BoardAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Normal

    Description

      We have a Cluster with 4 Servers in 2 Datacenters (2 in DC A and 2 in DC B), Root servers.
      We have a Replication Factor of 2, so atm we have 100% load on all 4 Servers. Around 250GB of Data. Everything works fine. Now we want to add 2 more Servers to the Cluster, one in each Datacenter. But we always get the same Kind of error while Bootstraping:

      ERROR 13:21:34 Unknown exception caught while attempting to update MaterializedView! messages_dump.messages
      java.lang.IllegalArgumentException: Mutation of 24032623 bytes is too large for the maxiumum size of 16777216
      at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:264) ~[apache-cassandra-3.0.5.jar:3.0.5]
      at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:469) [apache-cassandra-3.0.5.jar:3.0.5]
      at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) [apache-cassandra-3.0.5.jar:3.0.5]
      at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) [apache-cassandra-3.0.5.jar:3.0.5]
      at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) [apache-cassandra-3.0.5.jar:3.0.5]
      at org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager.java:146) ~[apache-cassandra-3.0.5.jar:3.0.5]
      at org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:724) ~[apache-cassandra-3.0.5.jar:3.0.5]
      at org.apache.cassandra.db.view.ViewManager.pushViewReplicaUpdates(ViewManager.java:149) ~[apache-cassandra-3.0.5.jar:3.0.5]
      at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:487) [apache-cassandra-3.0.5.jar:3.0.5]
      at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) [apache-cassandra-3.0.5.jar:3.0.5]
      at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) [apache-cassandra-3.0.5.jar:3.0.5]
      at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) [apache-cassandra-3.0.5.jar:3.0.5]
      at org.apache.cassandra.db.Mutation.applyUnsafe(Mutation.java:236) [apache-cassandra-3.0.5.jar:3.0.5]
      at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:169) [apache-cassandra-3.0.5.jar:3.0.5]
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_11]
      at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_11]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_11]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_11]
      at java.lang.Thread.run(Thread.java:745) [na:1.8.0_11]

      and

      WARN 13:21:34 Some data streaming failed. Use nodetool to check bootstrap state and resume. For more, see `nodetool help bootstrap`. IN_PROGRESS

      And if we Resume the Bootstrap it starts all over again and then it fails with the same Error Message.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned Assign to me
            aheiss Alexander HeiƟ
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment