Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-2036

Prevent out of memory in the master server, if the query result is large

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.11.1
    • Fix Version/s: 0.12.0, 0.11.1
    • Component/s: Web UI
    • Labels:
      None

      Description

      Tajo Master server is a no response below situation.

      Test SQL:
      select col1, col2, .... col100 from big_table
      (big_table has rows: 1,000,000,000)

      Web UI parameters:
      Limit : 10MB
      Rows/Page : 25

      Master Server Log:
      2016-01-06 10:35:59,973 INFO org.apache.tajo.tuple.memory.ResizableMemoryBlock: Increase DirectRowBlock to 64.0 KiB
      2016-01-06 10:36:00,739 INFO org.apache.tajo.tuple.memory.ResizableMemoryBlock: Increase DirectRowBlock to 64.0 KiB
      2016-01-06 10:36:01,316 INFO org.apache.tajo.master.exec.NonForwardQueryResultFileScanner: "Sent result to client for 7cba1175-e77b-4876-957d-2ff8722e940b, queryId: q_1452043069985_0001 rows: 3986313
      2016-01-06 10:36:49,561 INFO org.apache.tajo.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6385ms
      GC pool 'PS MarkSweep' had collection(s): count=1 time=5460ms
      GC pool 'PS Scavenge' had collection(s): count=1 time=993ms
      2016-01-06 10:37:03,102 WARN org.apache.tajo.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 10587ms
      GC pool 'PS MarkSweep' had collection(s): count=1 time=10669ms
      GC pool 'PS Scavenge' had collection(s): count=1 time=401ms

      Master Server Stacktrace:
      "pool-11-thread-1" #379 prio=5 os_prio=0 tid=0x00007f47d424d000 nid=0xe26c waiting on condition [0x00007f4834cfa000]
      java.lang.Thread.State: WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)

      • parking to wait for <0x000000076e9c7220> (a com.google.common.util.concurrent.AbstractFuture$Sync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:275)
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
        at org.apache.tajo.jdbc.FetchResultSet.nextTuple(FetchResultSet.java:77)
        at org.apache.tajo.jdbc.TajoResultSetBase.next(TajoResultSetBase.java:711)
        at org.apache.tajo.webapp.QueryExecutorServlet$QueryRunner.MakeResultText(QueryExecutorServlet.java:511)
        at org.apache.tajo.webapp.QueryExecutorServlet$QueryRunner.getQueryResult(QueryExecutorServlet.java:466)
        at org.apache.tajo.webapp.QueryExecutorServlet$QueryRunner.run(QueryExecutorServlet.java:347)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

        Activity

        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user combineads opened a pull request:

        https://github.com/apache/tajo/pull/928

        TAJO-2036: Prevent out of memory in the master server, if the query result is large

        This problem will occur frequently in our production server.
        Please confirm.

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/combineads/tajo TAJO-2036

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/928.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #928


        commit 09387b86df03cb1a53fd82da650038a8a0dd364a
        Author: Byunghwa Yun <combine@combineads.co.kr>
        Date: 2016-01-06T06:24:26Z

        TAJO-2036: Prevent out of memory in the master server, if the query
        result is large


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user combineads opened a pull request: https://github.com/apache/tajo/pull/928 TAJO-2036 : Prevent out of memory in the master server, if the query result is large This problem will occur frequently in our production server. Please confirm. You can merge this pull request into a Git repository by running: $ git pull https://github.com/combineads/tajo TAJO-2036 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/928.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #928 commit 09387b86df03cb1a53fd82da650038a8a0dd364a Author: Byunghwa Yun <combine@combineads.co.kr> Date: 2016-01-06T06:24:26Z TAJO-2036 : Prevent out of memory in the master server, if the query result is large
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on a diff in the pull request:

        https://github.com/apache/tajo/pull/928#discussion_r48938840

        — Diff: tajo-core/src/main/java/org/apache/tajo/webapp/QueryExecutorServlet.java —
        @@ -507,14 +508,20 @@ private void MakeResultText(ResultSet res, TableDesc desc) throws SQLException

        { numOfRows = resultRows; }

        + int currentResultSize = 0;
        int rowCount = 0;
        while (res.next()) {

        • if(rowCount > numOfRows) {
          + if(rowCount > numOfRows || currentResultSize > sizeLimit) {
            • End diff –

        ```numOfRows = (long)((float)(resultRows) * ((float)sizeLimit / (float) resultRows));``` on the 506 line looks not necessary. Would you remove it?

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/928#discussion_r48938840 — Diff: tajo-core/src/main/java/org/apache/tajo/webapp/QueryExecutorServlet.java — @@ -507,14 +508,20 @@ private void MakeResultText(ResultSet res, TableDesc desc) throws SQLException { numOfRows = resultRows; } + int currentResultSize = 0; int rowCount = 0; while (res.next()) { if(rowCount > numOfRows) { + if(rowCount > numOfRows || currentResultSize > sizeLimit) { End diff – ```numOfRows = (long)((float)(resultRows) * ((float)sizeLimit / (float) resultRows));``` on the 506 line looks not necessary. Would you remove it?
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user combineads commented on the pull request:

        https://github.com/apache/tajo/pull/928#issuecomment-171590158

        @jihoonson Anything else to commit this PR?

        Show
        githubbot ASF GitHub Bot added a comment - Github user combineads commented on the pull request: https://github.com/apache/tajo/pull/928#issuecomment-171590158 @jihoonson Anything else to commit this PR?
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on the pull request:

        https://github.com/apache/tajo/pull/928#issuecomment-171892339

        @combineads, sorry for late response.
        Actually, my intention was to keep both factors, but clean some unnecessary codes.
        I think we need to keep only the max number of rows rather than max byte size in the future. However, for now, it would be ok if we support both factors. How about adding the max number of rows again?

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/928#issuecomment-171892339 @combineads, sorry for late response. Actually, my intention was to keep both factors, but clean some unnecessary codes. I think we need to keep only the max number of rows rather than max byte size in the future. However, for now, it would be ok if we support both factors. How about adding the max number of rows again?
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user combineads commented on the pull request:

        https://github.com/apache/tajo/pull/928#issuecomment-173855449

        Hi @jihoonson .
        I did to add the row limit field.
        Thank you for review.

        Show
        githubbot ASF GitHub Bot added a comment - Github user combineads commented on the pull request: https://github.com/apache/tajo/pull/928#issuecomment-173855449 Hi @jihoonson . I did to add the row limit field. Thank you for review.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on a diff in the pull request:

        https://github.com/apache/tajo/pull/928#discussion_r51079765

        — Diff: tajo-core/src/main/java/org/apache/tajo/webapp/QueryExecutorServlet.java —
        @@ -197,7 +203,6 @@ public void service(HttpServletRequest request,
        errorResponse(response, queryRunner.error);
        return;
        }

        • returnValue.put("numOfRows", queryRunner.numOfRows);
            • End diff –

        limitRow also needs to be added to returnValue.

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/928#discussion_r51079765 — Diff: tajo-core/src/main/java/org/apache/tajo/webapp/QueryExecutorServlet.java — @@ -197,7 +203,6 @@ public void service(HttpServletRequest request, errorResponse(response, queryRunner.error); return; } returnValue.put("numOfRows", queryRunner.numOfRows); End diff – limitRow also needs to be added to returnValue.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on a diff in the pull request:

        https://github.com/apache/tajo/pull/928#discussion_r51079959

        — Diff: tajo-core/src/main/resources/webapps/admin/query_executor.jsp —
        @@ -87,6 +88,11 @@ function runQuery() {
        } else if(Math.ceil(Number($("#sizeLimit").val())) > 0)

        { SIZE_LIMIT = Number($("#sizeLimit").val()) * 1024 * 1024; }

        + if(Math.ceil(Number($("#rowLimit").val())) >= 10) {
        — End diff –

        IMO, we don't have to set the upper bound. It would be better to allow a very large value for rowLimit if users want.

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/928#discussion_r51079959 — Diff: tajo-core/src/main/resources/webapps/admin/query_executor.jsp — @@ -87,6 +88,11 @@ function runQuery() { } else if(Math.ceil(Number($("#sizeLimit").val())) > 0) { SIZE_LIMIT = Number($("#sizeLimit").val()) * 1024 * 1024; } + if(Math.ceil(Number($("#rowLimit").val())) >= 10) { — End diff – IMO, we don't have to set the upper bound. It would be better to allow a very large value for rowLimit if users want.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on a diff in the pull request:

        https://github.com/apache/tajo/pull/928#discussion_r51080021

        — Diff: tajo-core/src/main/java/org/apache/tajo/webapp/QueryExecutorServlet.java —
        @@ -197,7 +203,6 @@ public void service(HttpServletRequest request,
        errorResponse(response, queryRunner.error);
        return;
        }

        • returnValue.put("numOfRows", queryRunner.numOfRows);
            • End diff –

        Oh, my mistake. Never mind.

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/928#discussion_r51080021 — Diff: tajo-core/src/main/java/org/apache/tajo/webapp/QueryExecutorServlet.java — @@ -197,7 +203,6 @@ public void service(HttpServletRequest request, errorResponse(response, queryRunner.error); return; } returnValue.put("numOfRows", queryRunner.numOfRows); End diff – Oh, my mistake. Never mind.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user combineads commented on the pull request:

        https://github.com/apache/tajo/pull/928#issuecomment-175980187

        Hi @jihoonson .
        I did to remove the max row limit.
        Thank you for review.

        Show
        githubbot ASF GitHub Bot added a comment - Github user combineads commented on the pull request: https://github.com/apache/tajo/pull/928#issuecomment-175980187 Hi @jihoonson . I did to remove the max row limit. Thank you for review.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on the pull request:

        https://github.com/apache/tajo/pull/928#issuecomment-176009149

        +1 LGTM. This patch does not relate to unit tests.
        I'll commit shortly.

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/928#issuecomment-176009149 +1 LGTM. This patch does not relate to unit tests. I'll commit shortly.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/tajo/pull/928

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/928
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-CODEGEN-build #665 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/665/)
        TAJO-2036: Prevent out of memory in the master server, if the query (jihoonson: rev e9e7a36c7b1814755e22a83e81c8ab0604064517)

        • tajo-core/src/main/java/org/apache/tajo/webapp/QueryExecutorServlet.java
        • tajo-core/src/main/resources/webapps/admin/query_executor.jsp
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #665 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/665/ ) TAJO-2036 : Prevent out of memory in the master server, if the query (jihoonson: rev e9e7a36c7b1814755e22a83e81c8ab0604064517) tajo-core/src/main/java/org/apache/tajo/webapp/QueryExecutorServlet.java tajo-core/src/main/resources/webapps/admin/query_executor.jsp
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-build #1068 (See https://builds.apache.org/job/Tajo-master-build/1068/)
        TAJO-2036: Prevent out of memory in the master server, if the query (jihoonson: rev e9e7a36c7b1814755e22a83e81c8ab0604064517)

        • tajo-core/src/main/java/org/apache/tajo/webapp/QueryExecutorServlet.java
        • tajo-core/src/main/resources/webapps/admin/query_executor.jsp
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-build #1068 (See https://builds.apache.org/job/Tajo-master-build/1068/ ) TAJO-2036 : Prevent out of memory in the master server, if the query (jihoonson: rev e9e7a36c7b1814755e22a83e81c8ab0604064517) tajo-core/src/main/java/org/apache/tajo/webapp/QueryExecutorServlet.java tajo-core/src/main/resources/webapps/admin/query_executor.jsp
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-0.11.1-build #160 (See https://builds.apache.org/job/Tajo-0.11.1-build/160/)
        TAJO-2036: Prevent out of memory in the master server, if the query (jihoonson: rev fcc2215fb63653d3e7c7a57725e35ff6f997b77f)

        • tajo-core/src/main/java/org/apache/tajo/webapp/QueryExecutorServlet.java
        • tajo-core/src/main/resources/webapps/admin/query_executor.jsp
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-0.11.1-build #160 (See https://builds.apache.org/job/Tajo-0.11.1-build/160/ ) TAJO-2036 : Prevent out of memory in the master server, if the query (jihoonson: rev fcc2215fb63653d3e7c7a57725e35ff6f997b77f) tajo-core/src/main/java/org/apache/tajo/webapp/QueryExecutorServlet.java tajo-core/src/main/resources/webapps/admin/query_executor.jsp

          People

          • Assignee:
            combine Byunghwa Yun
            Reporter:
            combine Byunghwa Yun
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Due:
              Created:
              Updated:
              Resolved:

              Development