Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-665

sort buffer size must be dealt as long type values.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0, 0.9.0
    • Component/s: None
    • Labels:
      None

      Description

      sort buffer size (tajo.executor.external-sort.buffer-mb in TajoConf) is used to determine in-memory buffer for external merge sort. Currently, its type is integer. So, if users can set more than 2GB to sort buffer size, external merge operator will meet negative sort buffer size. Finally, it will cause significant performance degradation.

      1. TAJO-665.patch
        1 kB
        Hyoungjun Kim

        Activity

        Hide
        hjkim Hyoungjun Kim added a comment -

        I changed int to long in ExternalSortExec class.
        Please review this patch.

        Show
        hjkim Hyoungjun Kim added a comment - I changed int to long in ExternalSortExec class. Please review this patch.
        Hide
        hyunsik Hyunsik Choi added a comment -

        +1

        Thank you for quick fixing. I'll commit it.

        Show
        hyunsik Hyunsik Choi added a comment - +1 Thank you for quick fixing. I'll commit it.
        Hide
        tajoqa Tajo QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12633014/TAJO-665.patch
        against master revision 4089c9d.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

        +1 checkstyle. The patch generated 0 code style errors.

        -1 findbugs. The patch appears to introduce 184 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in tajo-core/tajo-core-backend.

        Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/193//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/193//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html
        Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/193//console

        This message is automatically generated.

        Show
        tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12633014/TAJO-665.patch against master revision 4089c9d. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 184 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-core/tajo-core-backend. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/193//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/193//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/193//console This message is automatically generated.
        Hide
        hyunsik Hyunsik Choi added a comment -

        Thank you for your contribution. committed the patch to master and branch-0.8.0.

        Show
        hyunsik Hyunsik Choi added a comment - Thank you for your contribution. committed the patch to master and branch-0.8.0.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-0.8.0-build #12 (See https://builds.apache.org/job/Tajo-0.8.0-build/12/)
        TAJO-665: sort buffer size must be dealt as long type values. (hyoungjunkim via hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=be1663bd17461e158120613938d82e37cf1e9380)

        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java
        • CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-0.8.0-build #12 (See https://builds.apache.org/job/Tajo-0.8.0-build/12/ ) TAJO-665 : sort buffer size must be dealt as long type values. (hyoungjunkim via hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=be1663bd17461e158120613938d82e37cf1e9380 ) tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-build #106 (See https://builds.apache.org/job/Tajo-master-build/106/)
        TAJO-665: sort buffer size must be dealt as long type values. (hyoungjunkim via hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=887252b25558710f03002c175723842cd75c17c2)

        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java
        • CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-build #106 (See https://builds.apache.org/job/Tajo-master-build/106/ ) TAJO-665 : sort buffer size must be dealt as long type values. (hyoungjunkim via hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=887252b25558710f03002c175723842cd75c17c2 ) tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #109 (See https://builds.apache.org/job/Tajo-master-build/109/)
        TAJO-665: sort buffer size must be dealt as long type values. (missed casting). (hyunsik: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=399c600c78a7e43d404fe2c7a69573800c26a727)

        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java
        • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #109 (See https://builds.apache.org/job/Tajo-master-build/109/ ) TAJO-665 : sort buffer size must be dealt as long type values. (missed casting). (hyunsik: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=399c600c78a7e43d404fe2c7a69573800c26a727 ) tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-0.8.0-build #15 (See https://builds.apache.org/job/Tajo-0.8.0-build/15/)
        TAJO-665: sort buffer size must be dealt as long type values. (missed casting). (hyunsik: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=835d3f691e564d68aa4b07a4bdf8a9ed8e68995c)

        • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-0.8.0-build #15 (See https://builds.apache.org/job/Tajo-0.8.0-build/15/ ) TAJO-665 : sort buffer size must be dealt as long type values. (missed casting). (hyunsik: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=835d3f691e564d68aa4b07a4bdf8a9ed8e68995c ) tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java

          People

          • Assignee:
            hjkim Hyoungjun Kim
            Reporter:
            hyunsik Hyunsik Choi
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development