Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-1906

Decrease the default size of hash map and array list for testing

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0, 0.11.1
    • Component/s: Unit Test
    • Labels:
      None

      Description

      The default size of hash maps and array lists for aggregation, sort, or join is at least 10000, but the number of tuples is much smaller that it in most of our tests.
      Thus, most spaces for those data structures are not used during tests. We need to reduce such unnecessary memory usage.

        Activity

        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user jihoonson opened a pull request:

        https://github.com/apache/tajo/pull/799

        TAJO-1906: Decrease the default size of hash map and array list for testing

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/jihoonson/tajo-2 TAJO-1906

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/799.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #799


        commit b6166ca361e57a8d412e56802cb274ea0a0aa79e
        Author: Jihoon Son <jihoonson@apache.org>
        Date: 2015-10-05T05:05:13Z

        TAJO-1906


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user jihoonson opened a pull request: https://github.com/apache/tajo/pull/799 TAJO-1906 : Decrease the default size of hash map and array list for testing You can merge this pull request into a Git repository by running: $ git pull https://github.com/jihoonson/tajo-2 TAJO-1906 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/799.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #799 commit b6166ca361e57a8d412e56802cb274ea0a0aa79e Author: Jihoon Son <jihoonson@apache.org> Date: 2015-10-05T05:05:13Z TAJO-1906
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on the pull request:

        https://github.com/apache/tajo/pull/799#issuecomment-145431579

        I also removed unused MemSortExec.

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/799#issuecomment-145431579 I also removed unused MemSortExec.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user hyunsik commented on the pull request:

        https://github.com/apache/tajo/pull/799#issuecomment-146074914

        The patch looks good to me. It would be great if we use ``JOIN_HASH_TABLE_SIZE`` without ``TEST_`` prefix.

        Show
        githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/799#issuecomment-146074914 The patch looks good to me. It would be great if we use ``JOIN_HASH_TABLE_SIZE`` without ``TEST_`` prefix.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/tajo/pull/799

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/799
        Hide
        jihoonson Jihoon Son added a comment -

        Committed to master and branch-0.11.0

        Show
        jihoonson Jihoon Son added a comment - Committed to master and branch-0.11.0
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-CODEGEN-build #577 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/577/)
        TAJO-1906: Decrease the default size of hash map and array list for (jihoonson: rev 3ca900d7a01b60f689d8ecd411e30b7e36620d80)

        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/BasicPhysicalExecutorVisitor.java
        • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/RightOuterMergeJoinExec.java
        • tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/PhysicalExecutorVisitor.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/CommonHashJoinExec.java
        • tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestProgressExternalSortExec.java
        • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/HashAggregateExec.java
        • tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestExternalSortExec.java
        • tajo-core-tests/src/test/resources/results/TestGroupByQuery/testPythonUdaf2.result
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java
        • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestJoinQuery.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/MemSortExec.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/DistinctGroupbyHashAggregationExec.java
        • tajo-core-tests/src/test/resources/results/TestTajoCli/testHelpSessionVars.result
        • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestGroupByQuery.java
        • CHANGES
        • tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestSortExec.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java
        • tajo-common/src/main/java/org/apache/tajo/SessionVars.java
        • tajo-core-tests/src/test/resources/results/TestGroupByQuery/testGroupBy2.result
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/MergeJoinExec.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/MergeFullOuterJoinExec.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #577 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/577/ ) TAJO-1906 : Decrease the default size of hash map and array list for (jihoonson: rev 3ca900d7a01b60f689d8ecd411e30b7e36620d80) tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/BasicPhysicalExecutorVisitor.java tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/RightOuterMergeJoinExec.java tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/PhysicalExecutorVisitor.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/CommonHashJoinExec.java tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestProgressExternalSortExec.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/HashAggregateExec.java tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestExternalSortExec.java tajo-core-tests/src/test/resources/results/TestGroupByQuery/testPythonUdaf2.result tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestJoinQuery.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/MemSortExec.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/DistinctGroupbyHashAggregationExec.java tajo-core-tests/src/test/resources/results/TestTajoCli/testHelpSessionVars.result tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestGroupByQuery.java CHANGES tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestSortExec.java tajo-core/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java tajo-common/src/main/java/org/apache/tajo/SessionVars.java tajo-core-tests/src/test/resources/results/TestGroupByQuery/testGroupBy2.result tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/MergeJoinExec.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/MergeFullOuterJoinExec.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #952 (See https://builds.apache.org/job/Tajo-master-build/952/)
        TAJO-1906: Decrease the default size of hash map and array list for (jihoonson: rev 3ca900d7a01b60f689d8ecd411e30b7e36620d80)

        • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/PhysicalExecutorVisitor.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/MergeJoinExec.java
        • CHANGES
        • tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestExternalSortExec.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java
        • tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestSortExec.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/CommonHashJoinExec.java
        • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestJoinQuery.java
        • tajo-core-tests/src/test/resources/results/TestGroupByQuery/testPythonUdaf2.result
        • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/MemSortExec.java
        • tajo-core-tests/src/test/resources/results/TestTajoCli/testHelpSessionVars.result
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/HashAggregateExec.java
        • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestGroupByQuery.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/MergeFullOuterJoinExec.java
        • tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestProgressExternalSortExec.java
        • tajo-core-tests/src/test/resources/results/TestGroupByQuery/testGroupBy2.result
        • tajo-common/src/main/java/org/apache/tajo/SessionVars.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/RightOuterMergeJoinExec.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/BasicPhysicalExecutorVisitor.java
        • tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/DistinctGroupbyHashAggregationExec.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #952 (See https://builds.apache.org/job/Tajo-master-build/952/ ) TAJO-1906 : Decrease the default size of hash map and array list for (jihoonson: rev 3ca900d7a01b60f689d8ecd411e30b7e36620d80) tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/PhysicalExecutorVisitor.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/MergeJoinExec.java CHANGES tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestExternalSortExec.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestSortExec.java tajo-core/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/CommonHashJoinExec.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestJoinQuery.java tajo-core-tests/src/test/resources/results/TestGroupByQuery/testPythonUdaf2.result tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/MemSortExec.java tajo-core-tests/src/test/resources/results/TestTajoCli/testHelpSessionVars.result tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/HashAggregateExec.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestGroupByQuery.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/MergeFullOuterJoinExec.java tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestProgressExternalSortExec.java tajo-core-tests/src/test/resources/results/TestGroupByQuery/testGroupBy2.result tajo-common/src/main/java/org/apache/tajo/SessionVars.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/RightOuterMergeJoinExec.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/BasicPhysicalExecutorVisitor.java tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/DistinctGroupbyHashAggregationExec.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-0.11.0-build #108 (See https://builds.apache.org/job/Tajo-0.11.0-build/108/)
        TAJO-1906: Decrease the default size of hash map and array list for (jihoonson: rev 1aa06796efa69ec40786fe3258803dd4e60fc7f7)

        • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestGroupByQuery.java
        • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/MergeFullOuterJoinExec.java
        • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestJoinQuery.java
        • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java
        • CHANGES
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/RightOuterMergeJoinExec.java
        • tajo-common/src/main/java/org/apache/tajo/SessionVars.java
        • tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestExternalSortExec.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/MergeJoinExec.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/BasicPhysicalExecutorVisitor.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java
        • tajo-core-tests/src/test/resources/results/TestTajoCli/testHelpSessionVars.result
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java
        • tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestSortExec.java
        • tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java
        • tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestProgressExternalSortExec.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/DistinctGroupbyHashAggregationExec.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/PhysicalExecutorVisitor.java
        • tajo-core-tests/src/test/resources/results/TestGroupByQuery/testGroupBy2.result
        • tajo-core-tests/src/test/resources/results/TestGroupByQuery/testPythonUdaf2.result
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/CommonHashJoinExec.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/MemSortExec.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/HashAggregateExec.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-0.11.0-build #108 (See https://builds.apache.org/job/Tajo-0.11.0-build/108/ ) TAJO-1906 : Decrease the default size of hash map and array list for (jihoonson: rev 1aa06796efa69ec40786fe3258803dd4e60fc7f7) tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestGroupByQuery.java tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/MergeFullOuterJoinExec.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestJoinQuery.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java CHANGES tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/RightOuterMergeJoinExec.java tajo-common/src/main/java/org/apache/tajo/SessionVars.java tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestExternalSortExec.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/MergeJoinExec.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/BasicPhysicalExecutorVisitor.java tajo-core/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java tajo-core-tests/src/test/resources/results/TestTajoCli/testHelpSessionVars.result tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestSortExec.java tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestProgressExternalSortExec.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/DistinctGroupbyHashAggregationExec.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/PhysicalExecutorVisitor.java tajo-core-tests/src/test/resources/results/TestGroupByQuery/testGroupBy2.result tajo-core-tests/src/test/resources/results/TestGroupByQuery/testPythonUdaf2.result tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/CommonHashJoinExec.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/MemSortExec.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/HashAggregateExec.java

          People

          • Assignee:
            jihoonson Jihoon Son
            Reporter:
            jihoonson Jihoon Son
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development