[HIVE-16233] llap: Query failed with AllocatorOutOfMemoryException

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: llap
• Target Version/s: 3.0.0
• Assignee: Sergey Shelukhin
• Reporter: Siddharth Seth
• Created: 16/Mar/17 18:24
• Updated: 03/May/19 01:11
• Resolved: 21/Jun/17 01:11

Description

TaskAttempt 5 failed, info=[Error: Error while running task ( failure ) : attempt_1488231257387_2288_25_05_000056_5:java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.io.IOException: org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException: Failed to allocate 262144; at 0 out of 1
        at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
        at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
        at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
        at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
        at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
        at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
        at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
        at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
        at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.io.IOException: org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException: Failed to allocate 262144; at 0 out of 1
        at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:74)
        at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)
        at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
        ... 15 more
Caused by: java.io.IOException: java.io.IOException: org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException: Failed to allocate 262144; at 0 out of 1
        at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
        at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
        at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
        at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
        at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
        at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
        at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
        at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
        at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:62)
        ... 17 more
Caused by: java.io.IOException: org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException: Failed to allocate 262144; at 0 out of 1
        at org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:425)
        at org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:413)
        at org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:235)
        at org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:232)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
        at org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:232)
        at org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:94)
        ... 6 more
Caused by: org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException: Failed to allocate 262144; at 0 out of 1
        at org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:275)
        at org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedStream(EncodedReaderImpl.java:675)
        at org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:412)
        ... 14 more

Also reported by selinazh while using older bits.

This JIRA will be used to track a long-term fix.
We were initially planning to write a different allocator with compaction, but now we are stuck with buddyallocator.
There are two approaches - compaction on top of buddy allocator (should be fairly simple actually, aside from indirection... luckily for the latter, we can take the buffer out in most places and we can reuse the existing locking-to-prevent-eviction mechanism to also prevent reallocation); or writing a better allocator. Probably the former is preferred, need to think about the latter.

Issue Links

• blocks: HIVE-15665 LLAP: OrcFileMetadata objects in cache can impact heap usage
• causes: HIVE-21686 Brute Force eviction can lead to a random uncontrolled eviction pattern.
• is related to: HIVE-16236 BuddyAllocator fragmentation - short-term fix
• is related to: HIVE-17176 Add ASF header for LlapAllocatorBuffer.java
• links to: RB