Description
Orc memory manager uses MemoryMX bean to get max heap usage and assumes 50% (configurable) is available for orc writers. This works well if container model where container size is close to Xmx. In LLAP for example, a single container runs multiple executors and a single task take up 50% of heap which is typically much more that the task executors own memory limit.
Example: 128Gb container + 32 executors (assuming no cache) means there is 4GB available per executor. But a single task that opens multiple orc writers in the default case will assume 64GB memory is available for orc when in fact it should only use 2GB from memory per executor.
WriterOption provides a mechanism to plug-in a custom memory manager for such scenarios which is greate. But if the custom memory manager want to extend MemoryManagerImpl class, MemoryManagerImpl should use getTotalMemoryPool() everywhere so that custom memory manager can just Override getTotalMemoryPool() method and rest of the code from MemoryManagerImpl.
Attachments
Issue Links
- is required by
-
HIVE-20649 LLAP aware memory manager for Orc writers
-
- Closed
-
- links to
