[IMPALA-2901] Impala query using 100% CPU system state

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: Impala 2.0
• Component/s: Backend
• Assignee: Unassigned
• Reporter: wangsen
• Created: 28/Jan/16 08:40
• Updated: 29/Jan/16 19:00
• Resolved: 29/Jan/16 19:00

Description

I'm using impala query a table，use nmon monitor cpu usage，i find the process impalad using 100% CPU system state， then i use jstack attach the impalad process and use
perf top tool monitor impalad process (reference attach files) ，log's detail is below

Thread 139715: (state = IN_VM)

• sun.misc.Unsafe.allocateMemory(long) @bci=0 (Interpreted frame)
• java.nio.DirectByteBuffer.<init>(int) @bci=49, line=127 (Interpreted frame)
• java.nio.ByteBuffer.allocateDirect(int) @bci=5, line=306 (Interpreted frame)
• org.apache.hadoop.hdfs.util.DirectBufferPool.getBuffer(int) @bci=61, line=70 (Interpreted frame)
• org.apache.hadoop.hdfs.BlockReaderLocal.createDataBufIfNeeded() @bci=20, line=244 (Interpreted frame)
• org.apache.hadoop.hdfs.BlockReaderLocal.fillDataBuf(boolean) @bci=1, line=434 (Interpreted frame)
• org.apache.hadoop.hdfs.BlockReaderLocal.readWithBounceBuffer(java.nio.ByteBuffer, boolean) @bci=147, line=504 (Interpreted frame)
• org.apache.hadoop.hdfs.BlockReaderLocal.read(java.nio.ByteBuffer) @bci=135, line=386 (Interpreted frame)
• org.apache.hadoop.hdfs.DFSInputStream$ByteBufferStrategy.doRead(org.apache.hadoop.hdfs.BlockReader, int, int, org.apache.hadoop.hdfs.DFSInputStream$ReadStatistics) @bci=26, line=706 (Interpreted frame)
• org.apache.hadoop.hdfs.DFSInputStream.readBuffer(org.apache.hadoop.hdfs.DFSInputStream$ReaderStrategy, int, int, java.util.Map) @bci=14, line=740 (Interpreted frame)
• org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(org.apache.hadoop.hdfs.DFSInputStream$ReaderStrategy, int, int) @bci=141, line=797 (Interpreted frame)
• org.apache.hadoop.hdfs.DFSInputStream.read(java.nio.ByteBuffer) @bci=16, line=845 (Interpreted frame)
• org.apache.hadoop.fs.FSDataInputStream.read(java.nio.ByteBuffer) @bci=18, line=144 (Interpreted frame)

Thread 139697: (state = IN_VM)

• sun.misc.Unsafe.allocateMemory(long) @bci=0 (Interpreted frame)
• java.nio.DirectByteBuffer.<init>(int) @bci=49, line=127 (Interpreted frame)
• java.nio.ByteBuffer.allocateDirect(int) @bci=5, line=306 (Interpreted frame)
• org.apache.hadoop.hdfs.util.DirectBufferPool.getBuffer(int) @bci=61, line=70 (Interpreted frame)
• org.apache.hadoop.hdfs.BlockReaderLocal.createDataBufIfNeeded() @bci=20, line=244 (Interpreted frame)
• org.apache.hadoop.hdfs.BlockReaderLocal.fillDataBuf(boolean) @bci=1, line=434 (Interpreted frame)
• org.apache.hadoop.hdfs.BlockReaderLocal.readWithBounceBuffer(java.nio.ByteBuffer, boolean) @bci=147, line=504 (Interpreted frame)
• org.apache.hadoop.hdfs.BlockReaderLocal.read(java.nio.ByteBuffer) @bci=135, line=386 (Interpreted frame)
• org.apache.hadoop.hdfs.DFSInputStream$ByteBufferStrategy.doRead(org.apache.hadoop.hdfs.BlockReader, int, int, org.apache.hadoop.hdfs.DFSInputStream$ReadStatistics) @bci=26, line=706 (Interpreted frame)
• org.apache.hadoop.hdfs.DFSInputStream.readBuffer(org.apache.hadoop.hdfs.DFSInputStream$ReaderStrategy, int, int, java.util.Map) @bci=14, line=740 (Interpreted frame)
• org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(org.apache.hadoop.hdfs.DFSInputStream$ReaderStrategy, int, int) @bci=141, line=797 (Interpreted frame)
• org.apache.hadoop.hdfs.DFSInputStream.read(java.nio.ByteBuffer) @bci=16, line=845 (Interpreted frame)
• org.apache.hadoop.fs.FSDataInputStream.read(java.nio.ByteBuffer) @bci=18, line=144 (Interpreted frame)

Issue Links

• duplicates: IMPALA-1722 Clients that don't Fetch can cause Impala to busy spin in ScannerThreads
• duplicates: IMPALA-1730 Reduce the window of spinning for Parquet and base-sequence scanners