[CASSANDRA-11727] Streaming error while Bootstraping Materialized View

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Fix Version/s: 3.0.5
• Component/s: Feature/Materialized Views, Legacy/Streaming and Messaging

Environment

Ubuntu 14.04
Oracle JDK 1.8.0_11
16GB RAM
Cassandra Version 3.0.5

• Severity: Normal
• Assignee: Unassigned
• Reporter: Alexander Heiß
• Created: 06/May/16 13:39
• Updated: 16/Apr/19 09:30
• Resolved: 06/May/16 13:53

Description

We have a Cluster with 4 Servers in 2 Datacenters (2 in DC A and 2 in DC B), Root servers.
We have a Replication Factor of 2, so atm we have 100% load on all 4 Servers. Around 250GB of Data. Everything works fine. Now we want to add 2 more Servers to the Cluster, one in each Datacenter. But we always get the same Kind of error while Bootstraping:

ERROR 13:21:34 Unknown exception caught while attempting to update MaterializedView! messages_dump.messages
java.lang.IllegalArgumentException: Mutation of 24032623 bytes is too large for the maxiumum size of 16777216
at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:264) ~[apache-cassandra-3.0.5.jar:3.0.5]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:469) [apache-cassandra-3.0.5.jar:3.0.5]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) [apache-cassandra-3.0.5.jar:3.0.5]
at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) [apache-cassandra-3.0.5.jar:3.0.5]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) [apache-cassandra-3.0.5.jar:3.0.5]
at org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager.java:146) ~[apache-cassandra-3.0.5.jar:3.0.5]
at org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:724) ~[apache-cassandra-3.0.5.jar:3.0.5]
at org.apache.cassandra.db.view.ViewManager.pushViewReplicaUpdates(ViewManager.java:149) ~[apache-cassandra-3.0.5.jar:3.0.5]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:487) [apache-cassandra-3.0.5.jar:3.0.5]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) [apache-cassandra-3.0.5.jar:3.0.5]
at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) [apache-cassandra-3.0.5.jar:3.0.5]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) [apache-cassandra-3.0.5.jar:3.0.5]
at org.apache.cassandra.db.Mutation.applyUnsafe(Mutation.java:236) [apache-cassandra-3.0.5.jar:3.0.5]
at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:169) [apache-cassandra-3.0.5.jar:3.0.5]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_11]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_11]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_11]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_11]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_11]

and

WARN 13:21:34 Some data streaming failed. Use nodetool to check bootstrap state and resume. For more, see `nodetool help bootstrap`. IN_PROGRESS

And if we Resume the Bootstrap it starts all over again and then it fails with the same Error Message.

Issue Links

• duplicates: CASSANDRA-11670 Rebuilding or streaming MV generates mutations larger than max_mutation_size_in_kb