[IGNITE-12096] Used memory metric does not contract when cache is emptied - FreeList REUSE_BUCKET is not accounted for

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.7
• Component/s: cache
• Ignite Flags: Docs Required
• Assignee: Unassigned
• Reporter: Colin Cassidy
• Created: 23/Aug/19 11:40
• Updated: 06/Jul/20 14:06

Description

When using the Ignite metrics API to measure available memory, the usage figures appear to be accurate while memory is being consumed - but when memory is freed the metrics do not drop. They appear to report that memory has not been freed up, even though it has.

A reliable estimate of memory consumption is very important for solutions that don't use native persistence - as this is the only controllable way of avoiding a critical OOM condition.

Reproducer below. This affects Ignite 2.7+.

public class MemoryTest2 { 

    private static final String CACHE_NAME = "cache"; 
    private static final String DEFAULT_MEMORY_REGION = "Default_Region"; 
    private static final long MEM_SIZE = 100L * 1024 * 1024; 

    @Test 
    public void testOOM() throws InterruptedException { 
        try (Ignite ignite = startIgnite("IgniteMemoryMonitorTest1")) { 
            fillDataRegion(ignite); 
            CacheConfiguration<Object, Object> cfg = new 
CacheConfiguration<>(CACHE_NAME); 
            cfg.setStatisticsEnabled(true); 
            IgniteCache<Object, Object> cache = 
ignite.getOrCreateCache(cfg); 

            // Clear all entries from the cache to free up memory 
            memUsed(ignite); 
            cache.clear(); 
            cache.removeAll(); 
            cache.put("Key", "Value"); 
            memUsed(ignite); 
            cache.destroy(); 
            Thread.sleep(5000); 

            // Should now report close to 0% but reports 59% still 
            memUsed(ignite); 
        } 
    } 
    
    private Ignite startIgnite(String instanceName) { 
        IgniteConfiguration cfg = new IgniteConfiguration(); 
        cfg.setIgniteInstanceName(instanceName); 
        cfg.setDataStorageConfiguration(createDataStorageConfiguration()); 
        cfg.setFailureHandler(new NoOpFailureHandler()); 
        return Ignition.start(cfg); 
    } 

    private DataStorageConfiguration createDataStorageConfiguration() { 
        return new DataStorageConfiguration() 
                .setDefaultDataRegionConfiguration( 
                        new DataRegionConfiguration() 
                                .setName(DEFAULT_MEMORY_REGION) 
                                .setInitialSize(MEM_SIZE) 
                                .setMaxSize(MEM_SIZE) 
                                .setMetricsEnabled(true)); 
    } 

    private void fillDataRegion(Ignite ignite) { 
        byte[] megabyte = new byte[1024 * 1024]; 
            IgniteCache<Object, Object> cache = 
                    ignite.getOrCreateCache(CACHE_NAME); 
            for (int i = 0; i < 50; i++) { 
                cache.put(i, megabyte); 
                memUsed(ignite); 
            } 
    } 

    private void memUsed(Ignite ignite) { 
        DataRegionConfiguration defaultDataRegionCfg = 
ignite.configuration() 
                .getDataStorageConfiguration() 
                .getDefaultDataRegionConfiguration(); 
        String regionName = defaultDataRegionCfg.getName(); 
        DataRegionMetrics metrics = ignite.dataRegionMetrics(regionName); 
        float usedMem = metrics.getPagesFillFactor() * 
metrics.getTotalAllocatedPages() * metrics.getPageSize(); 
        float pctUsed = 100 * usedMem / defaultDataRegionCfg.getMaxSize(); 
        System.out.println("Memory used: " + pctUsed + "%"); 
    } 
} 

Issue Links

• is caused by: IGNITE-10669 NPE in freelist.PagesList.findTailIndex
• is related to: IGNITE-11848 [IEP-35] Monitoring&Profiling. Phase 1