[CASSANDRA-13011] heap exhaustion when cleaning table with wide partitions and a secondary index attached to it

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Won't Fix
• Fix Version/s: 2.2.19
• Component/s: Feature/2i Index, Local/Compaction
• Severity: Normal
• Assignee: Unassigned
• Reporter: Milan Majercik
• Created: 07/Dec/16 07:14
• Updated: 01/Aug/21 12:47
• Resolved: 30/Jun/21 16:39

Description

We have a table with rather wide partitions and a secondary index attached to it. When tried to clean unused data on a node after expansion of our cluster via issuing nodetool cleanup command we observed a heap exhaustion issue. The culprit appears to be in method org.apache.cassandra.db.compaction.CompactionManager.CleanupStrategy.Full.cleanup as it tries to remove related secondary index entries. The method first populates a list will all cells belonging to the given partition...

                while (row.hasNext())
                {
                    OnDiskAtom column = row.next();

                    if (column instanceof Cell && cfs.indexManager.indexes((Cell) column))
                    {
                        if (indexedColumnsInRow == null)
                            indexedColumnsInRow = new ArrayList<>();

                        indexedColumnsInRow.add((Cell) column);
                    }
                }

... and then submits it to the index manager for removal.

                    // acquire memtable lock here because secondary index deletion may cause a race. See CASSANDRA-3712
                    try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start())
                    {
                        cfs.indexManager.deleteFromIndexes(row.getKey(), indexedColumnsInRow, opGroup);
                    }

After imposing a limit on array size and implementing some sort of pagination the cleanup worked fine.