[DRILL-5517] Provide size-aware set operations in value vectors

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.11.0
• Fix Version/s: 1.11.0
• Labels: ready-to-commit
• Assignee: Paul Rogers
• Reporter: Paul Rogers
• Reviewer: Karthikeyan Manivannan
• Created: 16/May/17 19:42
• Updated: 03/Jul/17 20:47
• Resolved: 03/Jul/17 20:47

Description

DRILL-5211 describes a memory fragmentation issue in Drill. The resolution is to limit vector sizes to 16 MB (the size of Netty memory allocation "slabs.") Effort starts by providing "size-aware" set operations in value vectors which:

• Operate as setSafe() while vectors are below 16 MB.
• Throw a new, specific exception (VectorOverflowException) if setting the value (and growing the vector) would exceed the vector limit.

The methods in value vectors then become the foundation on which we can construct size-aware record batch "writers."

Issue Links

• is part of: DRILL-5211 Queries fail due to direct memory fragmentation
• links to: GitHub Pull Request #840