Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-17473

sstables changing in snapshots

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Triage Needed
    • Normal
    • Resolution: Unresolved
    • None
    • None
    • None
    • All
    • None

    Description

      We use cassandra snapshots and tar to make full backups of our cassandra clusters. Sometimes, tar fails with a message like

      tar: data/addresses/addresses-eb0196100b7d11ec852b1541747d640a/snapshots/backup20220318183708/nb-167-big-Data.db: file changed as we read it

      This is kind of strange, since we're reading from a snapshot.

      The (very simplified) relevant snippet looks roughly like

      nice nodetool "${JMX_ARGS[@]}" snapshot -t "$TAG" "${KEYSPACES[@]}"
      tar --hard-dereference -czpf data///snapshots/"$TAG"/

      This happens maybe 1% of the time when taking backups.

      There are no concurrent snapshots going on, but there are concurrent compactions and repairs, of course. If it matters, this cluster is running incremental repairs.

      This is on Cassandra 4.0.3.

      It seems wrong to me that an sstable could ever be written to while it's in a snapshot.

      Attachments

        Activity

          People

            Unassigned Unassigned
            jbrownEP James Brown
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: