[PHOENIX-2685] SpoolingResultIterator opening a large number of files, hitting process max open file limit

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.5.1, 4.6.0, 4.7.0
• Labels: patch
• Flags: Patch
• Assignee: Unassigned
• Reporter: Alok Singh
• Created: 16/Feb/16 02:41
• Updated: 16/Feb/16 02:50

Description

During one of out load tests, in which we execute 5-10 concurrent requests over a few hours, we have found that eventually the jvm process runs out of file handles due to 1000s of tmp files (/tmp/ResultSpoolerxxxxxx.bin).

Upon further investigation, it looks like that under high concurrent load, with the default 'phoenix.query.spoolThresholdBytes' value, SpoolingResultIterator will almost always run into this issue.

final MemoryChunk chunk = mm.allocate(0, thresholdBytes);
long waitTime = System.currentTimeMillis() - startTime;
GLOBAL_MEMORY_WAIT_TIME.update(waitTime);
memoryMetrics.getMemoryWaitTimeMetric().change(waitTime);
DeferredFileOutputStream spoolTo = null;
try {
    // Can't be bigger than int, since it's the max of the above allocation
    int size = (int)chunk.getSize();
    spoolTo = new DeferredFileOutputStream(size, "ResultSpooler",".bin", new File(spoolDirectory)) {
        @Override
        protected void thresholdReached() throws IOException {
            try {
                super.thresholdReached();
            } finally {
                chunk.close();
            }
        }
    };

The memory chunks successfully created here will not be released until a GC cycle invokes the MemoryChunk finalizer. Since the default behavoir is to allocate the full "spoolThresholdBytes" and only resize the chunk after iterating over the scan results, if there is sufficient concurrent load the GlobablMemoryManager will run out of freememory and cause the SpoolingIterators to always spool to disk.

int size = (int)chunk.getSize();
spoolTo = new DeferredFileOutputStream(size, "ResultSpooler",".bin", new File(spoolDirectory))

The "size" ends up being 0, causing the "thresholdReached()" method to be invoked when an attempt is made to write to the DeferredFileouputStream.

One way to mitigate this is to reduce the size of "phoenix.query.spoolThresholdBytes". We ended up reducing it to 10Mb.

Another would be to use a "Memory Usage" aware outputstream that has a reference to the memory manager and switches to a spooling only when allocation request to the memory manager fail. A patch with a simple MemoryManager based outputstream/SpoolingResultIterator is attached.