Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      In general streaming is working much better in 2.0.x than before, but we still get occasional hanging stream sessions.

      One of the nodes, the "follower", throws IOException: Broken pipe, causing all streams to fail with the "initiator" node. But the initiator node still thinks its sending and receiving files from the follower, causing the streaming to hang forever.

      Relevant lines from the logs of the "follower" attached. There's nothing relevant in the logs on the initiator node. There are no indications of retry attempts.

      1. errlogs
        50 kB
        Jimmy Mårdell

        Activity

        Hide
        yukim Yuki Morishita added a comment -

        Do you set streaming_socket_timeout_in_ms in cassandra.yaml?
        Default is 0, which means no timeout.

        We may want to pick different default value.

        Show
        yukim Yuki Morishita added a comment - Do you set streaming_socket_timeout_in_ms in cassandra.yaml? Default is 0, which means no timeout. We may want to pick different default value.
        Hide
        yarin Jimmy Mårdell added a comment -

        Ohh, nope! Will try that, thanks!

        "nodetool netstats" also shows some progress of each stream. Would it make sense to have a timeout on not making progress for some time?

        Show
        yarin Jimmy Mårdell added a comment - Ohh, nope! Will try that, thanks! "nodetool netstats" also shows some progress of each stream. Would it make sense to have a timeout on not making progress for some time?
        Hide
        philipthompson Philip Thompson added a comment -

        Yuki Morishita, do we consider this a bug since stream timeout can be set, preventing hanging on failed repairs?

        Show
        philipthompson Philip Thompson added a comment - Yuki Morishita , do we consider this a bug since stream timeout can be set, preventing hanging on failed repairs?
        Hide
        yukim Yuki Morishita added a comment -

        Yes, and I'm working on CASSANDRA-8621 as well and that can be fix for this as well.

        Show
        yukim Yuki Morishita added a comment - Yes, and I'm working on CASSANDRA-8621 as well and that can be fix for this as well.

          People

          • Assignee:
            yukim Yuki Morishita
            Reporter:
            yarin Jimmy Mårdell
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development