Details
Major Description
I’m using the following code to initialize my map, I’m also trying to get a handle the amount of memory to allocate. Using the code below the memory starts off right around 3 GB as I would expect. However, it keeps increasing up to 7 GB. I’m trying to load a map of unique addresses with a count of the number of hits for each. I’m starting small and using only 5 million of the 270 million records. I’ve performed a dump from the database of all of the unique values and in CSV form it is only 2.5 GB, hence the 3 GB starting point for the DirectMemory cache.
AllocationPolicy allocationPolicy = new RoundRobinAllocationPolicy();
MemoryManagerService<Integer> memoryManager = new MemoryManagerServiceImpl<>( allocationPolicy, true );
CacheService<String, Integer> cache = new DirectMemory<String, Integer>().setMap(new ConcurrentHashMap<String, Pointer<Integer>>()).setMemoryManager(memoryManager).setNumberOfBuffers(3).setSize(Ram.Mb(1024)).setInitialCapacity(Ram.Mb(1024)).setConcurrencyLevel(4).newCacheService();
As I read an load the file, I get about 400K in and I get the following stacktrace, but the process continues and finally around 1.2 Million records processed I get the second stacktrace. I’m using the following code to retrieve the current count value and put the count back into the map. I’ve changed the code to only process 500K records, when I dump the map contents I have a Number of Entries equal to NULL. That part I don’t understand, since I’m setting either a 1 or a count + 1 value.
-
-
- NOTE *** Using JDK 7 Update 51 I do not get the stacktraces!!!
-
Integer count = cache.retrieve(temp);
if (count != null)
{
cache.put(temp, count + 1);
}
else
{
cache.put(temp, 1);
}
100000
200000
300000
400000
Exception in thread "Timer-0" java.lang.IllegalArgumentException: The buffer org.apache.directmemory.memory.allocator.MergingByteBufferAllocator$MergingNioMemoryBuffer@25015757 seems not to belong to this allocator
at org.apache.directmemory.memory.allocator.MergingByteBufferAllocator$MergingNioMemoryBuffer.free(MergingByteBufferAllocator.java:396)
at org.apache.directmemory.memory.allocator.MergingByteBufferAllocator.free(MergingByteBufferAllocator.java:157)
at org.apache.directmemory.memory.MemoryManagerServiceImpl.free(MemoryManagerServiceImpl.java:185)
at org.apache.directmemory.memory.AbstractMemoryManager.free(AbstractMemoryManager.java:149)
at org.apache.directmemory.memory.AbstractMemoryManager.collectLFU(AbstractMemoryManager.java:139)
at org.apache.directmemory.cache.CacheServiceImpl.collectLFU(CacheServiceImpl.java:260)
at org.apache.directmemory.cache.CacheServiceImpl$1.run(CacheServiceImpl.java:85)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
500000
600000
700000
800000
900000
1000000
1100000
1200000
java.lang.IllegalArgumentException: The buffer org.apache.directmemory.memory.allocator.MergingByteBufferAllocator$MergingNioMemoryBuffer@39e2ab21 seems not to belong to this allocator
at org.apache.directmemory.memory.allocator.MergingByteBufferAllocator$MergingNioMemoryBuffer.free(MergingByteBufferAllocator.java:396)
at org.apache.directmemory.memory.allocator.MergingByteBufferAllocator.free(MergingByteBufferAllocator.java:157)
at org.apache.directmemory.memory.MemoryManagerServiceImpl.free(MemoryManagerServiceImpl.java:185)
at org.apache.directmemory.cache.CacheServiceImpl.store(CacheServiceImpl.java:148)
at org.apache.directmemory.cache.CacheServiceImpl.put(CacheServiceImpl.java:118)
at org.apache.directmemory.cache.CacheServiceImpl.put(CacheServiceImpl.java:109)
at com.datamentors.CreateVoltDBLoadFileAndMaps.<init>(CreateMaps.java:150)
at com.datamentors.CreateVoltDBLoadFileAndMaps.main(CreateMaps.java:290)