Details

      Description

      BSTIndex writer collect the key and offsets for range shuffle. if keys is unique, collector needs large memory.
      In case of sorted dataset, dataset don’t need to keep in the memory. It should be written in the file immediately

      Here is the stack trace

      Thread 63932: (state = BLOCKED)
      - org.apache.tajo.storage.VTuple.isBlankOrNull(int) @bci=15, line=67 (Compiled frame; information may be imprecise)
      - org.apache.tajo.storage.BaseTupleComparator.compare(org.apache.tajo.storage.Tuple, org.apache.tajo.storage.Tuple) @bci=32, line=112 (Compiled frame)
      - org.apache.tajo.storage.BaseTupleComparator.compare(java.lang.Object, java.lang.Object) @bci=9, line=37 (Compiled frame)
      - java.util.TreeMap.getEntryUsingComparator(java.lang.Object) @bci=29, line=376 (Compiled frame)
      - java.util.TreeMap.getEntry(java.lang.Object) @bci=9, line=345 (Compiled frame)
      - java.util.TreeMap.containsKey(java.lang.Object) @bci=2, line=232 (Compiled frame)
      - org.apache.tajo.storage.index.bst.BSTIndex$BSTIndexWriter$KeyOffsetCollector.put(org.apache.tajo.storage.Tuple, long) @bci=5, line=263 (Compiled frame)
      - org.apache.tajo.storage.index.bst.BSTIndex$BSTIndexWriter.write(org.apache.tajo.storage.Tuple, long) @bci=88, line=143 (Compiled frame)
      - org.apache.tajo.engine.planner.physical.RangeShuffleFileWriteExec.next() @bci=78, line=108 (Compiled frame)
      - org.apache.tajo.worker.TaskImpl.run() @bci=99, line=402 (Interpreted frame)
      - org.apache.tajo.worker.TaskContainer.run() @bci=149, line=65 (Interpreted frame)
      - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511 (Interpreted frame)
      - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Interpreted frame)
      - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
      - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
      - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
      

        Activity

        Hide
        hudson Hudson added a comment -

        ABORTED: Integrated in Tajo-0.11.1-build #129 (See https://builds.apache.org/job/Tajo-0.11.1-build/129/)
        TAJO-2000: BSTIndex can cause OOM. (jhkim: rev 70f6c126416fdc43e57b26bc01b8271daf1c8bf4)

        • tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/index/TestBSTIndex.java
        • tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/index/TestSingleCSVFileBSTIndex.java
        • tajo-common/src/main/java/org/apache/tajo/tuple/memory/HeapTuple.java
        • tajo-common/src/main/java/org/apache/tajo/storage/BufferPool.java
        • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/index/bst/BSTIndex.java
        • tajo-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/StoreIndexExec.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/RangeShuffleFileWriteExec.java
        • tajo-common/src/main/java/org/apache/tajo/tuple/memory/UnSafeTuple.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/BSTIndexScanExec.java
        • tajo-core/src/main/java/org/apache/tajo/worker/TaskImpl.java
        • tajo-common/src/main/java/org/apache/tajo/util/FileUtil.java
        • CHANGES
        Show
        hudson Hudson added a comment - ABORTED: Integrated in Tajo-0.11.1-build #129 (See https://builds.apache.org/job/Tajo-0.11.1-build/129/ ) TAJO-2000 : BSTIndex can cause OOM. (jhkim: rev 70f6c126416fdc43e57b26bc01b8271daf1c8bf4) tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/index/TestBSTIndex.java tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/index/TestSingleCSVFileBSTIndex.java tajo-common/src/main/java/org/apache/tajo/tuple/memory/HeapTuple.java tajo-common/src/main/java/org/apache/tajo/storage/BufferPool.java tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/index/bst/BSTIndex.java tajo-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/StoreIndexExec.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/RangeShuffleFileWriteExec.java tajo-common/src/main/java/org/apache/tajo/tuple/memory/UnSafeTuple.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/BSTIndexScanExec.java tajo-core/src/main/java/org/apache/tajo/worker/TaskImpl.java tajo-common/src/main/java/org/apache/tajo/util/FileUtil.java CHANGES
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #1017 (See https://builds.apache.org/job/Tajo-master-build/1017/)
        TAJO-2000: BSTIndex can cause OOM. (remove open() in init) (jhkim: rev 17af809cc15c5cd43a4547d733955432f38c4c3d)

        • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/index/bst/BSTIndex.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #1017 (See https://builds.apache.org/job/Tajo-master-build/1017/ ) TAJO-2000 : BSTIndex can cause OOM. (remove open() in init) (jhkim: rev 17af809cc15c5cd43a4547d733955432f38c4c3d) tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/index/bst/BSTIndex.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #1016 (See https://builds.apache.org/job/Tajo-master-build/1016/)
        TAJO-2000: BSTIndex can cause OOM. (jhkim: rev ee6c2b5fe75753ea0b5b54e833e86e368b7ef3b2)

        • tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/index/TestSingleCSVFileBSTIndex.java
        • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/index/bst/BSTIndex.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/BSTIndexScanExec.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/StoreIndexExec.java
        • tajo-common/src/main/java/org/apache/tajo/tuple/memory/HeapTuple.java
        • tajo-common/src/main/java/org/apache/tajo/storage/BufferPool.java
        • tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/index/TestBSTIndex.java
        • tajo-core/src/main/java/org/apache/tajo/worker/TaskImpl.java
        • CHANGES
        • tajo-common/src/main/java/org/apache/tajo/tuple/memory/UnSafeTuple.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/RangeShuffleFileWriteExec.java
        • tajo-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java
        • tajo-common/src/main/java/org/apache/tajo/util/FileUtil.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #1016 (See https://builds.apache.org/job/Tajo-master-build/1016/ ) TAJO-2000 : BSTIndex can cause OOM. (jhkim: rev ee6c2b5fe75753ea0b5b54e833e86e368b7ef3b2) tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/index/TestSingleCSVFileBSTIndex.java tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/index/bst/BSTIndex.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/BSTIndexScanExec.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/StoreIndexExec.java tajo-common/src/main/java/org/apache/tajo/tuple/memory/HeapTuple.java tajo-common/src/main/java/org/apache/tajo/storage/BufferPool.java tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/index/TestBSTIndex.java tajo-core/src/main/java/org/apache/tajo/worker/TaskImpl.java CHANGES tajo-common/src/main/java/org/apache/tajo/tuple/memory/UnSafeTuple.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/RangeShuffleFileWriteExec.java tajo-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java tajo-common/src/main/java/org/apache/tajo/util/FileUtil.java
        Hide
        jhkim Jinho Kim added a comment -

        committed it
        Thanks

        Show
        jhkim Jinho Kim added a comment - committed it Thanks
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-CODEGEN-build #623 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/623/)
        TAJO-2000: BSTIndex can cause OOM. (remove open() in init) (jhkim: rev 17af809cc15c5cd43a4547d733955432f38c4c3d)

        • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/index/bst/BSTIndex.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #623 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/623/ ) TAJO-2000 : BSTIndex can cause OOM. (remove open() in init) (jhkim: rev 17af809cc15c5cd43a4547d733955432f38c4c3d) tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/index/bst/BSTIndex.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-CODEGEN-build #622 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/622/)
        TAJO-2000: BSTIndex can cause OOM. (jhkim: rev ee6c2b5fe75753ea0b5b54e833e86e368b7ef3b2)

        • CHANGES
        • tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/index/TestBSTIndex.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/RangeShuffleFileWriteExec.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/StoreIndexExec.java
        • tajo-common/src/main/java/org/apache/tajo/tuple/memory/HeapTuple.java
        • tajo-common/src/main/java/org/apache/tajo/util/FileUtil.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/BSTIndexScanExec.java
        • tajo-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java
        • tajo-core/src/main/java/org/apache/tajo/worker/TaskImpl.java
        • tajo-common/src/main/java/org/apache/tajo/storage/BufferPool.java
        • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/index/bst/BSTIndex.java
        • tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/index/TestSingleCSVFileBSTIndex.java
        • tajo-common/src/main/java/org/apache/tajo/tuple/memory/UnSafeTuple.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #622 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/622/ ) TAJO-2000 : BSTIndex can cause OOM. (jhkim: rev ee6c2b5fe75753ea0b5b54e833e86e368b7ef3b2) CHANGES tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/index/TestBSTIndex.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/RangeShuffleFileWriteExec.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/StoreIndexExec.java tajo-common/src/main/java/org/apache/tajo/tuple/memory/HeapTuple.java tajo-common/src/main/java/org/apache/tajo/util/FileUtil.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/BSTIndexScanExec.java tajo-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java tajo-core/src/main/java/org/apache/tajo/worker/TaskImpl.java tajo-common/src/main/java/org/apache/tajo/storage/BufferPool.java tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/index/bst/BSTIndex.java tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/index/TestSingleCSVFileBSTIndex.java tajo-common/src/main/java/org/apache/tajo/tuple/memory/UnSafeTuple.java
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/tajo/pull/892

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/892
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on the pull request:

        https://github.com/apache/tajo/pull/892#issuecomment-162408487

        +1 LGTM!

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/892#issuecomment-162408487 +1 LGTM!
        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user jinossy opened a pull request:

        https://github.com/apache/tajo/pull/892

        TAJO-2000: BSTIndex can cause OOM.

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/jinossy/tajo TAJO-2000

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/892.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #892


        commit 96041d55cefe1eaf0dcccab4ac452fdfb100d0f6
        Author: Jinho Kim <jhkim@apache.org>
        Date: 2015-12-02T07:49:34Z

        TAJO-2000: BSTIndex can cause OOM.


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user jinossy opened a pull request: https://github.com/apache/tajo/pull/892 TAJO-2000 : BSTIndex can cause OOM. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jinossy/tajo TAJO-2000 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/892.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #892 commit 96041d55cefe1eaf0dcccab4ac452fdfb100d0f6 Author: Jinho Kim <jhkim@apache.org> Date: 2015-12-02T07:49:34Z TAJO-2000 : BSTIndex can cause OOM.

          People

          • Assignee:
            jhkim Jinho Kim
            Reporter:
            jhkim Jinho Kim
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development