Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-2036

Prevent out of memory in the master server, if the query result is large

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.11.1
    • Fix Version/s: 0.12.0, 0.11.1
    • Component/s: Web UI
    • Labels:
      None

      Description

      Tajo Master server is a no response below situation.

      Test SQL:
      select col1, col2, .... col100 from big_table
      (big_table has rows: 1,000,000,000)

      Web UI parameters:
      Limit : 10MB
      Rows/Page : 25

      Master Server Log:
      2016-01-06 10:35:59,973 INFO org.apache.tajo.tuple.memory.ResizableMemoryBlock: Increase DirectRowBlock to 64.0 KiB
      2016-01-06 10:36:00,739 INFO org.apache.tajo.tuple.memory.ResizableMemoryBlock: Increase DirectRowBlock to 64.0 KiB
      2016-01-06 10:36:01,316 INFO org.apache.tajo.master.exec.NonForwardQueryResultFileScanner: "Sent result to client for 7cba1175-e77b-4876-957d-2ff8722e940b, queryId: q_1452043069985_0001 rows: 3986313
      2016-01-06 10:36:49,561 INFO org.apache.tajo.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6385ms
      GC pool 'PS MarkSweep' had collection(s): count=1 time=5460ms
      GC pool 'PS Scavenge' had collection(s): count=1 time=993ms
      2016-01-06 10:37:03,102 WARN org.apache.tajo.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 10587ms
      GC pool 'PS MarkSweep' had collection(s): count=1 time=10669ms
      GC pool 'PS Scavenge' had collection(s): count=1 time=401ms

      Master Server Stacktrace:
      "pool-11-thread-1" #379 prio=5 os_prio=0 tid=0x00007f47d424d000 nid=0xe26c waiting on condition [0x00007f4834cfa000]
      java.lang.Thread.State: WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)

      • parking to wait for <0x000000076e9c7220> (a com.google.common.util.concurrent.AbstractFuture$Sync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:275)
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
        at org.apache.tajo.jdbc.FetchResultSet.nextTuple(FetchResultSet.java:77)
        at org.apache.tajo.jdbc.TajoResultSetBase.next(TajoResultSetBase.java:711)
        at org.apache.tajo.webapp.QueryExecutorServlet$QueryRunner.MakeResultText(QueryExecutorServlet.java:511)
        at org.apache.tajo.webapp.QueryExecutorServlet$QueryRunner.getQueryResult(QueryExecutorServlet.java:466)
        at org.apache.tajo.webapp.QueryExecutorServlet$QueryRunner.run(QueryExecutorServlet.java:347)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

        Attachments

          Activity

            People

            • Assignee:
              combine Byunghwa Yun
              Reporter:
              combine Byunghwa Yun
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved: