[DRILL-2698] Add allocation policy for (direct memory) limits per-query-per-node

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.8.0
• Fix Version/s: Future
• Component/s: Execution - Flow
• Assignee: Unassigned
• Reporter: Chris Westin
• Created: 06/Apr/15 18:52
• Updated: 01/Oct/15 16:38

Description

We want the allocator to support a policy that limits memory allocation per-query-per-node. Current policy is to divide the available direct memory by the number of fragments running on a node, and limit each fragment to that. This new policy would count fragments that belong to the same query and are running on the same node together. So, if query A has 3 fragments running on a node, and query B has 5 fragments running on the same node, and there are no other queries with fragments on that node, then there are two queries running on the node, and their allocation limit is the available direct memory divided by two. (Under the original policy, which divides the available memory by the number of fragments, A would be allowed 3/8 of the memory, and B would be allowed 5/8 of the memory.)

Note that this policy should be a user-settable boot-time option; the original policy would still also be available as an option.

Issue Links

• depends upon: DRILL-1942 Improve off-heap memory usage tracking