[CASSANDRA-8248] Possible memory leak

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Component/s: Legacy/Streaming and Messaging
• Severity: Normal
• Assignee: Joshua McKenzie
• Reporter: Alexander Sterligov

Authors

Joshua McKenzie

Reviewers

Benedict Elliott Smith

• Created: 04/Nov/14 00:18
• Updated: 16/Apr/19 09:31
• Resolved: 12/Dec/14 17:50

Description

Sometimes during repair cassandra starts to consume more memory than expected.

Total amount of data on node is about 20GB.
Size of the data directory is 66GC because of snapshots.

Top reports:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
15724 loadbase  20   0  493g  55g  44g S   28 44.2   4043:24 java

At the /proc/15724/maps there are a lot of deleted file maps

7f63a6102000-7f63a6332000 r--s 00000000 08:21 9442763 /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db (deleted)
7f63a6332000-7f63a6562000 r--s 00000000 08:21 9442763 /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db (deleted)
7f63a6562000-7f63a6792000 r--s 00000000 08:21 9442763 /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db (deleted)
7f63a6792000-7f63a69c2000 r--s 00000000 08:21 9442763 /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db (deleted)
7f63a69c2000-7f63a6bf2000 r--s 00000000 08:21 9442763 /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db (deleted)
7f63a6bf2000-7f63a6e22000 r--s 00000000 08:21 9442763 /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db (deleted)
7f63a6e22000-7f63a7052000 r--s 00000000 08:21 9442763 /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db (deleted)
7f63a7052000-7f63a7282000 r--s 00000000 08:21 9442763 /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db (deleted)
7f63a7282000-7f63a74b2000 r--s 00000000 08:21 9442763 /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db (deleted)
7f63a74b2000-7f63a76e2000 r--s 00000000 08:21 9442763 /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db (deleted)
7f63a76e2000-7f63a7912000 r--s 00000000 08:21 9442763 /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db (deleted)
7f63a7912000-7f63a7b42000 r--s 00000000 08:21 9442763 /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db (deleted)
7f63a7b42000-7f63a7d72000 r--s 00000000 08:21 9442763 /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db (deleted)
7f63a7d72000-7f63a7fa2000 r--s 00000000 08:21 9442763 /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db (deleted)
7f63a7fa2000-7f63a81d2000 r--s 00000000 08:21 9442763 /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db (deleted)
7f63a81d2000-7f63a8402000 r--s 00000000 08:21 9442763 /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db (deleted)
7f63a8402000-7f63a8622000 r--s 00000000 08:21 9442763 /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db (deleted)
7f63a8622000-7f63a8842000 r--s 00000000 08:21 9442763 /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db (deleted)
7f63a8842000-7f63a8a62000 r--s 00000000 08:21 9442763 /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db (deleted)
7f63a8a62000-7f63a8c82000 r--s 00000000 08:21 9442763 /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db (deleted)
7f63a8c82000-7f63a8ea2000 r--s 00000000 08:21 9442763 /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db (deleted)
7f63a8ea2000-7f63a90c2000 r--s 00000000 08:21 9442763 /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db (deleted)
7f63a90c2000-7f63a92e2000 r--s 00000000 08:21 9442763 /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db (deleted)

$ sudo grep deleted /proc/15724/maps | wc -l
640118

$ sudo grep -v deleted /proc/15724/maps | wc -l
303340

Issue Links

• duplicates: CASSANDRA-8061 tmplink files are not removed
• is broken by: CASSANDRA-6916 Preemptive opening of compaction result
• relates to: CASSANDRA-8061 tmplink files are not removed