[CASSANDRA-10117] FD Leak with DTCS

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Not A Problem
• Labels: dtcs
• Severity: Normal
• Assignee: Benedict Elliott Smith Assign to me
• Reporter: Philip Thompson

Authors

Benedict Elliott Smith

• Created: 18/Aug/15 17:09
• Updated: 16/Apr/19 09:30
• Resolved: 19/Aug/15 11:22

Description

Using 2.1-HEAD, specifically commit 972ae147247a, I am experiencing issues in a one node test with DTCS. These seem separate from CASSANDRA-9882.

Using an ec2 i2.2xlarge node with all default settings and the following schema:

CREATE KEYSPACE ks WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};

  CREATE TABLE tab (
      key uuid,
      year int,
      month int,
      day int,
      c0 blob,
      c1 blob,
      c2 blob,
      c3 blob,
      c4 blob,
      c5 blob,
      c6 blob,
      c7 blob,
      c8 blob,
      PRIMARY KEY ((year, month, day), key)
  ) WITH compaction = {'class': 'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy'};

I loaded 4500M rows via stress, which totaled ~1.2TB. I then performed a few mixed workloads via stress, which were 50% insert, 50% the following read: Select * from tab where year = ? and month = ? and day = ? limit 1000.

This was done to reproduce a separate issue for a user. I then received reports from the user that they were experiencing open FD counts per sstable in the thousands. With absolutely no load on my cluster, I was finding that any sstable with open FDs had between 243 and 245 open FDs. I then started a stress process performing the same read/write workload as before. I was then immediately seeing FD counts as high as 6615 for a single sstable.

I was determining FD counts per sstable with the following [example] call:
lsof | grep '16119-Data.db' | wc -l.

I still have this cluster running, for you to examine. System.log is attached.