Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-12905

Retry acquire MV lock on failure instead of throwing WTE on streaming

Agile BoardAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Critical

    Description

      Hello,

      I performed two upgrades to the current cluster (currently 15 nodes, 1 DC, private VLAN),
      first it was 2.2.5.1 and repair worked flawlessly,
      second upgrade was to 3.0.9 (with upgradesstables) and also repair worked well,
      then i upgraded 2 weeks ago to 3.9 - and the repair problems started.

      there are several errors types from the system.log (different nodes) :

      • Sync failed between /xxx.xxx.xxx.xxx and /xxx.xxx.xxx.xxx
      • Streaming error occurred on session with peer xxx.xxx.xxx.xxx Operation timed out - received only 0 responses
      • Remote peer xxx.xxx.xxx.xxx failed stream session
      • Session completed with the following error
        org.apache.cassandra.streaming.StreamException: Stream failed

      i use 3.9 default configuration with the cluster settings adjustments (3 seeds, GossipingPropertyFileSnitch).
      streaming_socket_timeout_in_ms is the default (86400000).

      i'm afraid from consistency problems while i'm not performing repair.

      Any ideas?

      Thanks,
      Nir.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            brstgt Benjamin Roth Assign to me
            zilkanir Nir Zilka
            Benjamin Roth
            Paulo Motta (Deprecated)
            Votes:
            0 Vote for this issue
            Watchers:
            14 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment