Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-10801

Unexplained inconsistent data with Cassandra 2.1

Agile BoardAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Cannot Reproduce
    • 2.1.x
    • None
    • None
    • Normal

    Description

      We are experiencing weird behavior which we cannot explain.

      We have a CF, with RF=3, and we are writing and reading data to it with consistency level of ONE.

      For some reason, we see inconsistent results when querying for data.
      Even for rows that were written a day ago, we're seeing inconsistent results (1 replca has the data, the two others don't).

      Now, I would expect to see timeouts/dropped mutations, but all relevant counters are not advancing, and I would also expect hints to fix this inconsistency within minutes, but yet it doesn't.

      cqlsh:testing> SELECT WRITETIME(last_update),site_id, tree_id, individual_id, last_update FROM testcf WHERE site_id = 229673621 AND tree_id = 9 AND individual_id = 9032483;            
      
       writetime(last_update) | site_id   | tree_id | individual_id | last_update
      ------------------------+-----------+---------+---------------+-------------
             1448988343028000 | 229673621 |       9 |       9032483 |  1380912397
      
      (1 rows)
      cqlsh:testing> SELECT WRITETIME(last_update),site_id, tree_id, individual_id, last_update FROM testcf WHERE site_id = 229673621 AND tree_id = 9 AND individual_id = 9032483;
      
      site_id   | tree_id | individual_id | last_update
      -----------+---------+---------------+-------------
      
      (0 rows)
      cqlsh:testing> SELECT dateof(now()) FROM system.local ;
      
       dateof(now())
      --------------------------
       2015-12-02 14:48:44+0000
      
      (1 rows)
      
      

      We are running with Cassandra 2.1.11 with Oracle Java 1.8.0_65-b17

      Attachments

        1. tracing.log
          41 kB
          Maor Cohen
        2. trace2.log
          32 kB
          Maor Cohen
        3. missing_rows.png
          49 kB
          Maor Cohen

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned Assign to me
            imriz Imri Zvik
            Votes:
            5 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment