[TAJO-2036] Prevent out of memory in the master server, if the query result is large

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.11.1
• Fix Version/s: 0.12.0, 0.11.1
• Component/s: Web UI
• Assignee: Byunghwa Yun
• Reporter: Byunghwa Yun
• Due: 07/Jan/16
• Created: 06/Jan/16 03:27
• Updated: 28/Jan/16 07:00
• Resolved: 28/Jan/16 06:56

Description

Tajo Master server is a no response below situation.

Test SQL:
select col1, col2, .... col100 from big_table
(big_table has rows: 1,000,000,000)

Web UI parameters:
Limit : 10MB
Rows/Page : 25

Master Server Log:
2016-01-06 10:35:59,973 INFO org.apache.tajo.tuple.memory.ResizableMemoryBlock: Increase DirectRowBlock to 64.0 KiB
2016-01-06 10:36:00,739 INFO org.apache.tajo.tuple.memory.ResizableMemoryBlock: Increase DirectRowBlock to 64.0 KiB
2016-01-06 10:36:01,316 INFO org.apache.tajo.master.exec.NonForwardQueryResultFileScanner: "Sent result to client for 7cba1175-e77b-4876-957d-2ff8722e940b, queryId: q_1452043069985_0001 rows: 3986313
2016-01-06 10:36:49,561 INFO org.apache.tajo.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6385ms
GC pool 'PS MarkSweep' had collection(s): count=1 time=5460ms
GC pool 'PS Scavenge' had collection(s): count=1 time=993ms
2016-01-06 10:37:03,102 WARN org.apache.tajo.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 10587ms
GC pool 'PS MarkSweep' had collection(s): count=1 time=10669ms
GC pool 'PS Scavenge' had collection(s): count=1 time=401ms

Master Server Stacktrace:
"pool-11-thread-1" #379 prio=5 os_prio=0 tid=0x00007f47d424d000 nid=0xe26c waiting on condition [0x00007f4834cfa000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x000000076e9c7220> (a com.google.common.util.concurrent.AbstractFuture$Sync)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
  at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:275)
  at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
  at org.apache.tajo.jdbc.FetchResultSet.nextTuple(FetchResultSet.java:77)
  at org.apache.tajo.jdbc.TajoResultSetBase.next(TajoResultSetBase.java:711)
  at org.apache.tajo.webapp.QueryExecutorServlet$QueryRunner.MakeResultText(QueryExecutorServlet.java:511)
  at org.apache.tajo.webapp.QueryExecutorServlet$QueryRunner.getQueryResult(QueryExecutorServlet.java:466)
  at org.apache.tajo.webapp.QueryExecutorServlet$QueryRunner.run(QueryExecutorServlet.java:347)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)