Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-1713

Change the type of edge cache in JoinGraphContext from HashMap to LRUMap

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: Planner/Optimizer
    • Labels:
      None

      Description

      Since the current implementation using HashMap involves the potential OOM with a lot of join tables. We should prevent it.

        Activity

        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user jihoonson opened a pull request:

        https://github.com/apache/tajo/pull/654

        TAJO-1713: Change the type of edge cache in JoinGraphContext from HashMap to LRUMap

        I've also cleared variables which are unnecessary after join optimization.

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/jihoonson/tajo-2 TAJO-1713

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/654.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #654


        commit 587ecbbc4d1e099c41abbe59908263140feeced8
        Author: Jihoon Son <jihoonson@apache.org>
        Date: 2015-07-25T15:03:01Z

        TAJO-1713


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user jihoonson opened a pull request: https://github.com/apache/tajo/pull/654 TAJO-1713 : Change the type of edge cache in JoinGraphContext from HashMap to LRUMap I've also cleared variables which are unnecessary after join optimization. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jihoonson/tajo-2 TAJO-1713 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/654.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #654 commit 587ecbbc4d1e099c41abbe59908263140feeced8 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-07-25T15:03:01Z TAJO-1713
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user hyunsik commented on the pull request:

        https://github.com/apache/tajo/pull/654#issuecomment-125184470

        LRUMap has different initial sizes according to whether it is used in tests or not. Is it necessary or is it enough for tests?

        Show
        githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/654#issuecomment-125184470 LRUMap has different initial sizes according to whether it is used in tests or not. Is it necessary or is it enough for tests?
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on the pull request:

        https://github.com/apache/tajo/pull/654#issuecomment-125191306

        It is enough for tests. According to my experience, the cache size is increased up to about 120 entries while inner joining 5 tables. So, I think it is enough to set the max cache size as 100 during tests.

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/654#issuecomment-125191306 It is enough for tests. According to my experience, the cache size is increased up to about 120 entries while inner joining 5 tables. So, I think it is enough to set the max cache size as 100 during tests.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on the pull request:

        https://github.com/apache/tajo/pull/654#issuecomment-125900011

        I've removed the codes for adjusting map size during tests because the overhead of map size seems not significant.

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/654#issuecomment-125900011 I've removed the codes for adjusting map size during tests because the overhead of map size seems not significant.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user hyunsik commented on the pull request:

        https://github.com/apache/tajo/pull/654#issuecomment-125969322

        LGTM! +1

        Show
        githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/654#issuecomment-125969322 LGTM! +1
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on the pull request:

        https://github.com/apache/tajo/pull/654#issuecomment-126147894

        Thanks for the review!

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/654#issuecomment-126147894 Thanks for the review!
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/tajo/pull/654

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/654
        Hide
        jihoonson Jihoon Son added a comment -

        Committed to master

        Show
        jihoonson Jihoon Son added a comment - Committed to master
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-CODEGEN-build #419 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/419/)
        TAJO-1713: Change the type of edge cache in JoinGraphContext from HashMap to LRUMap. (jihoonson: rev 97e61e6f4e9ff1ebeac388dd22e139e1fd486e99)

        • tajo-plan/src/main/java/org/apache/tajo/plan/LogicalOptimizer.java
        • tajo-common/src/main/java/org/apache/tajo/util/graph/SimpleDirectedGraph.java
        • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestBSTIndexExec.java
        • tajo-plan/src/main/java/org/apache/tajo/plan/joinorder/JoinGraphContext.java
        • CHANGES
        • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestSortExec.java
        • tajo-plan/src/main/java/org/apache/tajo/plan/joinorder/JoinGraph.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #419 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/419/ ) TAJO-1713 : Change the type of edge cache in JoinGraphContext from HashMap to LRUMap. (jihoonson: rev 97e61e6f4e9ff1ebeac388dd22e139e1fd486e99) tajo-plan/src/main/java/org/apache/tajo/plan/LogicalOptimizer.java tajo-common/src/main/java/org/apache/tajo/util/graph/SimpleDirectedGraph.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestBSTIndexExec.java tajo-plan/src/main/java/org/apache/tajo/plan/joinorder/JoinGraphContext.java CHANGES tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestSortExec.java tajo-plan/src/main/java/org/apache/tajo/plan/joinorder/JoinGraph.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-build #781 (See https://builds.apache.org/job/Tajo-master-build/781/)
        TAJO-1713: Change the type of edge cache in JoinGraphContext from HashMap to LRUMap. (jihoonson: rev 97e61e6f4e9ff1ebeac388dd22e139e1fd486e99)

        • tajo-plan/src/main/java/org/apache/tajo/plan/LogicalOptimizer.java
        • CHANGES
        • tajo-common/src/main/java/org/apache/tajo/util/graph/SimpleDirectedGraph.java
        • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestSortExec.java
        • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestBSTIndexExec.java
        • tajo-plan/src/main/java/org/apache/tajo/plan/joinorder/JoinGraphContext.java
        • tajo-plan/src/main/java/org/apache/tajo/plan/joinorder/JoinGraph.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-build #781 (See https://builds.apache.org/job/Tajo-master-build/781/ ) TAJO-1713 : Change the type of edge cache in JoinGraphContext from HashMap to LRUMap. (jihoonson: rev 97e61e6f4e9ff1ebeac388dd22e139e1fd486e99) tajo-plan/src/main/java/org/apache/tajo/plan/LogicalOptimizer.java CHANGES tajo-common/src/main/java/org/apache/tajo/util/graph/SimpleDirectedGraph.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestSortExec.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestBSTIndexExec.java tajo-plan/src/main/java/org/apache/tajo/plan/joinorder/JoinGraphContext.java tajo-plan/src/main/java/org/apache/tajo/plan/joinorder/JoinGraph.java

          People

          • Assignee:
            Unassigned
            Reporter:
            jihoonson Jihoon Son
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development