DRILL-5211 describes a memory fragmentation issue in Drill. The resolution is to limit vector sizes to 16 MB (the size of Netty memory allocation "slabs.") Effort starts by providing "size-aware" set operations in value vectors which:
- Operate as setSafe() while vectors are below 16 MB.
- Throw a new, specific exception (VectorOverflowException) if setting the value (and growing the vector) would exceed the vector limit.
The methods in value vectors then become the foundation on which we can construct size-aware record batch "writers."