[CASSANDRA-10801] Unexplained inconsistent data with Cassandra 2.1

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Cannot Reproduce
• Fix Version/s: 2.1.x
• Severity: Normal
• Assignee: Unassigned
• Reporter: Imri Zvik
• Created: 02/Dec/15 14:51
• Updated: 16/Apr/19 09:30
• Resolved: 22/Jul/16 16:26

Description

We are experiencing weird behavior which we cannot explain.

We have a CF, with RF=3, and we are writing and reading data to it with consistency level of ONE.

For some reason, we see inconsistent results when querying for data.
Even for rows that were written a day ago, we're seeing inconsistent results (1 replca has the data, the two others don't).

Now, I would expect to see timeouts/dropped mutations, but all relevant counters are not advancing, and I would also expect hints to fix this inconsistency within minutes, but yet it doesn't.

cqlsh:testing> SELECT WRITETIME(last_update),site_id, tree_id, individual_id, last_update FROM testcf WHERE site_id = 229673621 AND tree_id = 9 AND individual_id = 9032483;            

 writetime(last_update) | site_id   | tree_id | individual_id | last_update
------------------------+-----------+---------+---------------+-------------
       1448988343028000 | 229673621 |       9 |       9032483 |  1380912397

(1 rows)
cqlsh:testing> SELECT WRITETIME(last_update),site_id, tree_id, individual_id, last_update FROM testcf WHERE site_id = 229673621 AND tree_id = 9 AND individual_id = 9032483;

site_id   | tree_id | individual_id | last_update
-----------+---------+---------------+-------------

(0 rows)
cqlsh:testing> SELECT dateof(now()) FROM system.local ;

 dateof(now())
--------------------------
 2015-12-02 14:48:44+0000

(1 rows)

We are running with Cassandra 2.1.11 with Oracle Java 1.8.0_65-b17