[CASSANDRA-12886] Streaming failed due to SSL Socket connection reset

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.2.10, 3.0.12
• Severity: Normal
• Since Version: 3.7
• Assignee: Paulo Motta
• Reporter: Bing Wu

Authors

Paulo Motta

Reviewers

Yuki Morishita

• Created: 08/Nov/16 20:37
• Updated: 16/Apr/19 09:30
• Resolved: 02/Mar/17 01:38

Description

While running "nodetool repair", I see many instances of "javax.net.ssl.SSLException: java.net.SocketException: Connection reset" in system.logs on some nodes in the cluster. Timestamps correspond to streaming source/initiator's error messages of "sync failed between ..."

Setup:

• Cassandra 3.7.01
• CentOS 6.7 in AWS (multi-region)
• JDK version:

  java version "1.8.0_102"
  Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
  Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)
  

• cassandra.yaml:

  server_encryption_options:
      internode_encryption: all
      keystore: [path]
      keystore_password: [password]
      truststore: [path]
      truststore_password: [password]
      # More advanced defaults below:
      # protocol: TLS
      # algorithm: SunX509
      # store_type: JKS
      # cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
      require_client_auth: false
  

Error messages in system.log on the target host:

ERROR [STREAM-OUT-/54.247.111.232:7001] 2016-11-07 07:30:56,475 StreamSession.java:529 - [Stream #e14abcb0-a4bb-11e6-9758-55b9ac38b78e] Streaming error occurred on session with peer 54.247.111.232
javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset
        at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1541) ~[na:1.8.0_102]
        at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1553) ~[na:1.8.0_102]
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71) ~[na:1.8.0_102]
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[na:1.8.0_102]
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[na:1.8.0_102]
        at org.apache.cassandra.io.util.WrappedDataOutputStreamPlus.flush(WrappedDataOutputStreamPlus.java:66) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:371) [apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:342) [apache-cassandra-3.7.0.jar:3.7.0]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
Caused by: javax.net.ssl.SSLException: java.net.SocketException: Connection reset