[CASSANDRA-13821] A lot of ReadTimeoutException errors

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Not A Bug
• Fix Version/s: 2.2.5
• Component/s: Legacy/Core, Legacy/CQL
• Labels: performance windows

Environment

We have a Cassandra Cluster with 2 Datacenters:

Cassandra Version: 2.2.5

DC1 - OperationalDC:
3 nodes Virtual Server on VMWare.
S.O.: Windows 2012 Server 64bits-native
vCPU 12 Cores @ 2.5Ghz
RAM: 16GB
HDD: 3,6 TB Virtual Drive

DC2 - AnalyticalDC:
3 nodes Virtual Server on VMWare.
S.O.: Running on Windows 2012 Server 64bits-native
vCPU 12 Cores @ 2.5Ghz
RAM: 16GB
HDD: 3,6 TB Virtual Drive

• Severity: Normal
• Since Version: 2.2.5
• Assignee: Unassigned
• Reporter: Ramiro Rioboo
• Created: 29/Aug/17 22:34
• Updated: 16/Apr/19 09:30
• Resolved: 29/Aug/17 23:07

Description

Our application connects directly to OperationalDC via CQL.
During the daily operation we have a large number of WriteTimeoutException and ReadTimeoutException errors.

In the logs of Cassandra we can see that the read and write timeout errors are also recorded with the error:

Java.lang.RuntimeException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 1 responses.

Log Detail, Application Level:
_22:54:22,945 [ERROR] AbstractEntityManager - Insert object 'Event

{...}

' error com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency LOCAL_QUORUM (2 replica were required but only 1 acknowledged the write)_

Log Detail, Cassandra Level:
EBUG [SharedPool-Worker-6] 2017-08-24 08:36:40,114 StorageProxy.java:1893 - Range slice failure; received 0 of 1 responses for range 1 of 1 java.lang.RuntimeException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 1 responses.

The timeout error occurs 2 seconds later because of the configuration in the yaml:underlined text
_# How long the coordinator should wait for writes to complete
write_request_timeout_in_ms: 2000_