[CASSANDRA-12905] Retry acquire MV lock on failure instead of throwing WTE on streaming

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 3.0.11, 3.10
• Component/s: Feature/Materialized Views, Legacy/Streaming and Messaging

Environment

centos 6.7 x86_64

• Severity: Critical
• Assignee: Benjamin Roth
• Reporter: Nir Zilka

Authors

Benjamin Roth

Reviewers

Paulo Motta (Deprecated)

• Created: 12/Nov/16 18:53
• Updated: 16/Apr/19 09:30
• Resolved: 15/Dec/16 19:30

Description

Hello,

I performed two upgrades to the current cluster (currently 15 nodes, 1 DC, private VLAN),
first it was 2.2.5.1 and repair worked flawlessly,
second upgrade was to 3.0.9 (with upgradesstables) and also repair worked well,
then i upgraded 2 weeks ago to 3.9 - and the repair problems started.

there are several errors types from the system.log (different nodes) :

• Sync failed between /xxx.xxx.xxx.xxx and /xxx.xxx.xxx.xxx
• Streaming error occurred on session with peer xxx.xxx.xxx.xxx Operation timed out - received only 0 responses
• Remote peer xxx.xxx.xxx.xxx failed stream session
• Session completed with the following error
  org.apache.cassandra.streaming.StreamException: Stream failed

---

i use 3.9 default configuration with the cluster settings adjustments (3 seeds, GossipingPropertyFileSnitch).
streaming_socket_timeout_in_ms is the default (86400000).

i'm afraid from consistency problems while i'm not performing repair.

Any ideas?

Thanks,
Nir.

Issue Links

• breaks: CASSANDRA-13056 dtest failure in materialized_views_test.TestMaterializedViews.base_replica_repair_test
• relates to: CASSANDRA-12888 Incremental repairs broken for MVs and CDC
• relates to: CASSANDRA-12689 All MutationStage threads blocked, kills server
• relates to: CASSANDRA-10779 Mutations do not block for completion under view lock contention
• relates to: CASSANDRA-13299 Potential OOMs and lock contention in write path streams