[CASSANDRA-6298] Duplicate hard links; Cannot take snapshot (1.2)

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Not A Problem

Environment

Linux, multiple data directories, Java 6

• Severity: Normal
• Since Version: 1.2.9
• Assignee: Unassigned
• Reporter: Elias Ross
• Created: 04/Nov/13 22:06
• Updated: 16/Apr/19 09:32
• Resolved: 29/Jul/14 18:18

Description

I've been seeing this. It could be related to CASSANDRA-6093 in some way. The version is 1.2.9.

Caused by: java.lang.RuntimeException: Tried to create duplicate hard link to /data05/rhq/data/system/NodeIdInfo/snapshots/1383588365276/system-NodeIdInfo-ic-
1-TOC.txt
at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:70)
at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:1081)
at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1567)
at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1612)
at org.apache.cassandra.db.Table.snapshot(Table.java:194)
at org.apache.cassandra.service.StorageService.takeSnapshot(StorageService.java:2203)

$ ./nodetool -p 7299 snapshot
Requested creating snapshot for: all keyspaces
Exception in thread "main" java.lang.RuntimeException: Tried to create duplicate hard link to /data06/rhq/data/system/NodeIdInfo/snapshots/1383602745114/system-NodeIdInfo-ic-1-Summary.db

data:

/data02/rhq/data/system/peers/snapshots
/data02/rhq/data/system/peers/snapshots/1383602745114
/data02/rhq/data/system/peers/snapshots/1383602745114/system-peers-ic-33-Data.db
/data02/rhq/data/system/peers/snapshots/1383602745114/system-peers-ic-33-Statistics.db
/data02/rhq/data/system/peers/snapshots/1383602745114/system-peers-ic-33-TOC.txt
/data02/rhq/data/system/peers/snapshots/1383602745114/system-peers-ic-33-Summary.db
/data02/rhq/data/system/peers/snapshots/1383602745114/system-peers-ic-32-Filter.db
/data02/rhq/data/system/peers/snapshots/1383602745114/system-peers-ic-32-Summary.db
/data02/rhq/data/system/peers/snapshots/1383602745114/system-peers-ic-32-Data.db
/data02/rhq/data/system/peers/snapshots/1383602745114/system-peers-ic-33-Digest.sha1
/data02/rhq/data/system/peers/snapshots/1383602745114/system-peers-ic-32-TOC.txt
/data02/rhq/data/system/peers/snapshots/1383602745114/system-peers-ic-33-Index.db
/data02/rhq/data/system/peers/snapshots/1383602745114/system-peers-ic-32-Index.db
/data02/rhq/data/system/peers/snapshots/1383602745114/system-peers-ic-32-Digest.sha1
/data02/rhq/data/system/peers/snapshots/1383602745114/system-peers-ic-32-Statistics.db
/data02/rhq/data/system/peers/snapshots/1383602745114/system-peers-ic-33-Filter.db
/data02/rhq/data/system/local/snapshots
/data02/rhq/data/system/local/snapshots/1383602745114
/data02/rhq/data/system/local/snapshots/1383602745114/system-local-ic-58-TOC.txt
/data02/rhq/data/system/local/snapshots/1383602745114/system-local-ic-58-Digest.sha1
/data02/rhq/data/system/local/snapshots/1383602745114/system-local-ic-58-Index.db
/data02/rhq/data/system/local/snapshots/1383602745114/system-local-ic-58-Summary.db
/data02/rhq/data/system/local/snapshots/1383602745114/system-local-ic-58-Statistics.db
/data02/rhq/data/system/local/snapshots/1383602745114/system-local-ic-58-Filter.db
/data02/rhq/data/system/local/snapshots/1383602745114/system-local-ic-58-Data.db
/data03/rhq/data/system/local/snapshots
/data03/rhq/data/system/local/snapshots/1383602745114
/data03/rhq/data/system/local/snapshots/1383602745114/system-local-ic-57-Index.db
/data03/rhq/data/system/local/snapshots/1383602745114/system-local-ic-57-Statistics.db
/data03/rhq/data/system/local/snapshots/1383602745114/system-local-ic-57-Digest.sha1
/data03/rhq/data/system/local/snapshots/1383602745114/system-local-ic-57-TOC.txt
/data03/rhq/data/system/local/snapshots/1383602745114/system-local-ic-57-Summary.db
/data03/rhq/data/system/local/snapshots/1383602745114/system-local-ic-57-Filter.db
/data03/rhq/data/system/local/snapshots/1383602745114/system-local-ic-57-Data.db
/data05/rhq/data/system/NodeIdInfo/snapshots
/data05/rhq/data/system/NodeIdInfo/snapshots/1383602745114
/data05/rhq/data/system/NodeIdInfo/snapshots/1383602745114/system-NodeIdInfo-ic-1-Data.db
/data05/rhq/data/system/NodeIdInfo/snapshots/1383602745114/system-NodeIdInfo-ic-1-Statistics.db
/data05/rhq/data/system/NodeIdInfo/snapshots/1383602745114/system-NodeIdInfo-ic-1-Digest.sha1
/data05/rhq/data/system/NodeIdInfo/snapshots/1383602745114/system-NodeIdInfo-ic-1-Filter.db
/data05/rhq/data/system/NodeIdInfo/snapshots/1383602745114/system-NodeIdInfo-ic-1-TOC.txt
/data05/rhq/data/system/NodeIdInfo/snapshots/1383602745114/system-NodeIdInfo-ic-1-Index.db
/data05/rhq/data/system/NodeIdInfo/snapshots/1383602745114/system-NodeIdInfo-ic-1-Summary.db
/data06/rhq/data/system/NodeIdInfo/snapshots
/data06/rhq/data/system/NodeIdInfo/snapshots/1383602745114
/data06/rhq/data/system/NodeIdInfo/snapshots/1383602745114/system-NodeIdInfo-ic-1-Data.db
/data06/rhq/data/system/NodeIdInfo/snapshots/1383602745114/system-NodeIdInfo-ic-1-Statistics.db
/data06/rhq/data/system/NodeIdInfo/snapshots/1383602745114/system-NodeIdInfo-ic-1-Digest.sha1
/data06/rhq/data/system/NodeIdInfo/snapshots/1383602745114/system-NodeIdInfo-ic-1-Filter.db
/data06/rhq/data/system/NodeIdInfo/snapshots/1383602745114/system-NodeIdInfo-ic-1-TOC.txt
/data06/rhq/data/system/NodeIdInfo/snapshots/1383602745114/system-NodeIdInfo-ic-1-Index.db
/data06/rhq/data/system/NodeIdInfo/snapshots/1383602745114/system-NodeIdInfo-ic-1-Summary.db