[DRILL-5211] Queries fail due to direct memory fragmentation

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Paul Rogers
• Reporter: Paul Rogers
• Created: 22/Jan/17 00:21
• Updated: 26/Oct/17 04:43

Description

Consider a test of the external sort as follows:

• Direct memory: 3GB
• Input file: 18 GB, with one Varchar column of 8K width

The sort runs, spilling to disk. Once all data arrives, the sort beings to merge the results. But, to do that, it must first do an intermediate merge. For example, in this sort, there are 190 spill files, but only 19 can be merged at a time. (Each merge file contains 128 MB batches, and only 19 can fit in memory, giving a total footprint of 2.5 GB, well below the 3 GB limit.

Yet, when loading batch xx, Drill fails with an OOM error. At that point, total available direct memory is 3,817,865,216. (Obtained from maxMemory in the Bits class in the JDK.)

It appears that Drill wants to allocate 58,257,868 bytes, but the totalCapacity (again in Bits) is already 3,800,769,206, causing an OOM.

The problem is that, at this point, the external sort should not ask the system for more memory. The allocator for the external sort is at just 1,192,350,366 before the allocation request. Plenty of spare memory should be available, released when the in-memory batches were spilled to disk prior to merging. Indeed, earlier in the run, the sort had reached a peak memory usage of 2,710,716,416 bytes. This memory should be available for reuse during merging, and is plenty sufficient to fill the particular request in question.

Issue Links

• contains: DRILL-5453 Managed External Sort : Sorting on a lot of columns is taking unreasonably long time
• contains: DRILL-5486 Compliant text reader tries, but fails, to ignore empty rows
• contains: DRILL-5805 External Sort runs out of memory
• incorporates: DRILL-5517 Provide size-aware set operations in value vectors
• incorporates: DRILL-5657 Implement size-aware result set loader
• incorporates: DRILL-5688 Add repeated map support to column accessors
• incorporates: DRILL-5512 Standardize error handling in ScanBatch
• incorporates: DRILL-5514 Enhance VectorContainer to merge two row sets
• links to: GitHub Pull Request #866