[CASSANDRA-8559] OOM caused by large tombstone warning.

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.0.15, 2.1.5
• Labels: tombstone

Environment

2.0.11 / 2.1

• Severity: Normal
• Since Version: 2.0.2
• Assignee: Aleksey Yeschenko
• Reporter: Dominic Letz

Authors

Aleksey Yeschenko

Reviewers

Sam Tunnicliffe

• Created: 05/Jan/15 05:19
• Updated: 16/Apr/19 09:31
• Resolved: 22/Mar/15 11:35

Description

When running with high amount of tombstones the error message generation from CASSANDRA-6117 can lead to out of memory situation with the default setting.

Attached a heapdump viewed in visualvm showing how this construct created two 777mb strings to print the error message for a read query and then crashed OOM.

        if (respectTombstoneThresholds() && columnCounter.ignored() > DatabaseDescriptor.getTombstoneWarnThreshold())
        {
            StringBuilder sb = new StringBuilder();
            CellNameType type = container.metadata().comparator;
            for (ColumnSlice sl : slices)
            {
                assert sl != null;

                sb.append('[');
                sb.append(type.getString(sl.start));
                sb.append('-');
                sb.append(type.getString(sl.finish));
                sb.append(']');
            }

            logger.warn("Read {} live and {} tombstoned cells in {}.{} (see tombstone_warn_threshold). {} columns was requested, slices={}, delInfo={}",
                        columnCounter.live(), columnCounter.ignored(), container.metadata().ksName, container.metadata().cfName, count, sb, container.deletionInfo());
        }

Issue Links

• is related to: CASSANDRA-6117 Avoid death-by-tombstone by default