[CASSANDRA-4741] Cassandra stops persisting or flushing data

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Fix Version/s: 1.1.6

Environment

Linux, Windows
Cassandra 1.1.4 or 1.1.5

• Severity: Normal
• Assignee: Unassigned
• Reporter: Mauri Melato
• Created: 01/Oct/12 18:32
• Updated: 16/Apr/19 09:32
• Resolved: 11/Oct/12 13:19

Description

Hello,

we have a Cassandra instance running on a single node with Windows XP 32bit.
After some time of normal working and some restart, Cassandra seems to stop to persist data into the keyspace without raising any error.
It's possible to update/insert rows or columns and you can query and see the updated keyspace but if you stop and start Cassandra, you get a keyspace 'from the past', always from the same point in the past.

We were not able to understand how to trigger this inconsistency in the first place but once you get it, you can't get rid of it if not removing the data directories.
Here in attachment you can find a complete set of Cassandra data directories:

   cassandra-data/
   |-- commitlog
   |-- data
   `-- saved_caches

with which it's possible test the inconsistent scenario.
We tried this on another clean Win Xp 32 machine and we could get the same issue from the first run.
On a Win XP 64 and the same dir set, Cassandra didn't get into this issue, but from the log we saw a difference in the disk access mode:

• XP/64: DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap" instead of 'standard'
• XP/32: DiskAccessMode 'auto' determined to be standard, indexAccessMode is standard

Not sure this is the key factor, nevertheless in another test bed we experimented the same issue even on a WinXP 64 node but we couldn't collect the dataset.

Here a short cassandra-cli history to show the issue:

[default@NEUTRO_Keyspace] connect 192.168.1.33/9160;
Connected to: "NEUTRO Cluster" on 192.168.1.33/9160

[default@unknown] use neutro_keyspace;
Authenticated to keyspace: NEUTRO_Keyspace

[default@NEUTRO_Keyspace] describe;
Keyspace: NEUTRO_Keyspace:
  Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
  Durable Writes: true
    Options: [replication_factor:1]
  Column Families:
    ColumnFamily: TaskId
      Key Validation Class: org.apache.cassandra.db.marshal.AsciiType
      Default column value validator: org.apache.cassandra.db.marshal.UTF8Type
      Columns sorted by: org.apache.cassandra.db.marshal.AsciiType
      GC grace seconds: 864000
      Compaction min/max thresholds: 4/32
      Read repair chance: 1.0
      DC Local Read repair chance: 0.0
      Replicate on write: true
      Caching: KEYS_ONLY
      Bloom Filter FP chance: default
      Built indexes: []
      Column Metadata:
        Column Name: taskId
          Validation Class: org.apache.cassandra.db.marshal.LongType
      Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy
      Compression Options:
        sstable_compression: org.apache.cassandra.io.compress.SnappyCompressor

[default@NEUTRO_Keyspace] list TaskId;
Using default limit of 100
Using default column limit of 100
-------------------
RowKey: NE_MASTER_TASK_ID
=> (column=taskId, value=47, timestamp=1349090424494000)

1 Row Returned.
Elapsed time: 93 msec(s).

[default@NEUTRO_Keyspace] set  taskid['NE_MASTER_TASK_ID']['taskId']='55';
Value inserted.
Elapsed time: 31 msec(s).

[default@NEUTRO_Keyspace] list TaskId;
Using default limit of 100
Using default column limit of 100
-------------------
RowKey: NE_MASTER_TASK_ID
=> (column=taskId, value=55, timestamp=1349108488164000)

1 Row Returned.
Elapsed time: 31 msec(s).

...Stop and Start Cassandra...

[default@NEUTRO_Keyspace] connect 192.168.1.33/9160;
Connected to: "NEUTRO Cluster" on 192.168.1.33/9160

[default@unknown] use neutro_keyspace;
Authenticated to keyspace: NEUTRO_Keyspace

[default@NEUTRO_Keyspace] list TaskId;
Using default limit of 100
Using default column limit of 100
-------------------
RowKey: NE_MASTER_TASK_ID
=> (column=taskId, value=47, timestamp=1349090424494000)

1 Row Returned.
Elapsed time: 110 msec(s).

Thanks for your attention