Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-537

After TAJO-522, still OutOfMemoryError: unable to create new native thread

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: distributed query plan
    • Labels:
      None

      Description

      I ran a big query applied this patch, still OOM. This is because if the quantity of different connections is large, Hyoungjun's patch won't help. Different connection create different thread pool.
      Actually, we can share the thread pool among all the clients.

      1. TAJO-537.patch
        8 kB
        Min Zhou
      2. TAJO-537-v2.patch
        8 kB
        Min Zhou

        Issue Links

          Activity

          Hide
          hyunsik Hyunsik Choi added a comment -

          Thank you Min for your contribution.

          Show
          hyunsik Hyunsik Choi added a comment - Thank you Min for your contribution.
          Hide
          hyunsik Hyunsik Choi added a comment -

          committed it to master branch.

          Show
          hyunsik Hyunsik Choi added a comment - committed it to master branch.
          Hide
          coderplay Min Zhou added a comment -

          Sure. Let's do it through another ticket. Actually, I still see over 300 threads in worker's process if I run a big query. But the number will drop later. Seems it's due to the thread pool is a cached pool.

          Show
          coderplay Min Zhou added a comment - Sure. Let's do it through another ticket. Actually, I still see over 300 threads in worker's process if I run a big query. But the number will drop later. Seems it's due to the thread pool is a cached pool.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #22 (See https://builds.apache.org/job/Tajo-master-build/22/)
          TAJO-537: After TAJO-522, still OutOfMemoryError: unable to create new native thread. (Min Zhou via hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=e57cf426d8d1d1b924a32a10f81a8912f2b0c45f)

          • tajo-rpc/src/main/java/org/apache/tajo/rpc/AsyncRpcClient.java
          • CHANGES.txt
          • tajo-rpc/src/main/java/org/apache/tajo/rpc/BlockingRpcClient.java
          • tajo-rpc/src/main/java/org/apache/tajo/rpc/NettyClientBase.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryInProgress.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #22 (See https://builds.apache.org/job/Tajo-master-build/22/ ) TAJO-537 : After TAJO-522 , still OutOfMemoryError: unable to create new native thread. (Min Zhou via hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=e57cf426d8d1d1b924a32a10f81a8912f2b0c45f ) tajo-rpc/src/main/java/org/apache/tajo/rpc/AsyncRpcClient.java CHANGES.txt tajo-rpc/src/main/java/org/apache/tajo/rpc/BlockingRpcClient.java tajo-rpc/src/main/java/org/apache/tajo/rpc/NettyClientBase.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryInProgress.java
          Hide
          hyunsik Hyunsik Choi added a comment -

          +1

          The patch looks pretty good for me. I'll commit.

          There is one suggestion that can be dealt in a separate jira issue. It would be great to move the factory construction outside from the NettyClientBase. Then, factory release becomes possible when a JVM shutdowns.

          Show
          hyunsik Hyunsik Choi added a comment - +1 The patch looks pretty good for me. I'll commit. There is one suggestion that can be dealt in a separate jira issue. It would be great to move the factory construction outside from the NettyClientBase. Then, factory release becomes possible when a JVM shutdowns.
          Hide
          coderplay Min Zhou added a comment -

          Cleaner patch.

          Show
          coderplay Min Zhou added a comment - Cleaner patch.
          Hide
          coderplay Min Zhou added a comment -

          I tested it in a real cluster. This patch worked well.

          Show
          coderplay Min Zhou added a comment - I tested it in a real cluster. This patch worked well.

            People

            • Assignee:
              coderplay Min Zhou
              Reporter:
              coderplay Min Zhou
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development