Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-1716

Repartitioner.makeEvenDistributedFetchImpl() does not distribute fetches evenly

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: distributed query plan
    • Labels:
      None

      Description

      Repartitioner.makeEvenDistributedFetchImpl() is to be designed to create partitions with balancing the fetch size of each partition. Currently, it does not create partitions evenly due to a trivial bug.

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #776 (See https://builds.apache.org/job/Tajo-master-build/776/)
        TAJO-1716: Repartitioner.makeEvenDistributedFetchImpl() does not distribute fetches evenly. (jihoonson: rev 2950d8ec60fa769819b60e7e516387259b1491da)

        • tajo-core/src/test/java/org/apache/tajo/master/TestRepartitioner.java
        • tajo-core/src/main/java/org/apache/tajo/querymaster/Repartitioner.java
        • CHANGES
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #776 (See https://builds.apache.org/job/Tajo-master-build/776/ ) TAJO-1716 : Repartitioner.makeEvenDistributedFetchImpl() does not distribute fetches evenly. (jihoonson: rev 2950d8ec60fa769819b60e7e516387259b1491da) tajo-core/src/test/java/org/apache/tajo/master/TestRepartitioner.java tajo-core/src/main/java/org/apache/tajo/querymaster/Repartitioner.java CHANGES
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-CODEGEN-build #413 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/413/)
        TAJO-1716: Repartitioner.makeEvenDistributedFetchImpl() does not distribute fetches evenly. (jihoonson: rev 2950d8ec60fa769819b60e7e516387259b1491da)

        • tajo-core/src/test/java/org/apache/tajo/master/TestRepartitioner.java
        • tajo-core/src/main/java/org/apache/tajo/querymaster/Repartitioner.java
        • CHANGES
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #413 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/413/ ) TAJO-1716 : Repartitioner.makeEvenDistributedFetchImpl() does not distribute fetches evenly. (jihoonson: rev 2950d8ec60fa769819b60e7e516387259b1491da) tajo-core/src/test/java/org/apache/tajo/master/TestRepartitioner.java tajo-core/src/main/java/org/apache/tajo/querymaster/Repartitioner.java CHANGES
        Hide
        jihoonson Jihoon Son added a comment -

        Committed to master

        Show
        jihoonson Jihoon Son added a comment - Committed to master
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on the pull request:

        https://github.com/apache/tajo/pull/659#issuecomment-125871239

        Thanks for your review!

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/659#issuecomment-125871239 Thanks for your review!
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/tajo/pull/659

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/659
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jinossy commented on the pull request:

        https://github.com/apache/tajo/pull/659#issuecomment-125865448

        Big +1!!

        Show
        githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/659#issuecomment-125865448 Big +1!!
        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user jihoonson opened a pull request:

        https://github.com/apache/tajo/pull/659

        TAJO-1716: Repartitioner.makeEvenDistributedFetchImpl() does not distribute fetches evenly

        Here is the evaluation result.

        • # of samples: 1000
        • # of fetches: 2000
            1. Before patch
              ```
        1. of partitions: 10
          std dev: 7239.203108077597
        2. of partitions: 20
          std dev: 2507.946123823235
        3. of partitions: 30
          std dev: 1337.669359578645
        4. of partitions: 40
          std dev: 847.1299428068963
        5. of partitions: 50
          std dev: 590.6100520648041
        6. of partitions: 60
          std dev: 441.75456483838553
        7. of partitions: 70
          std dev: 340.5741442844647
        8. of partitions: 80
          std dev: 270.92110936581224
        9. of partitions: 90
          std dev: 221.5819175222533
        10. of partitions: 100
          std dev: 184.4234215060522
          ```
            1. After patch
              ```
        11. of partitions: 10
          13.61763562654033
        12. of partitions: 20
          21.6693331706813
        13. of partitions: 30
          15.134361197689614
        14. of partitions: 40
          13.04185569631777
        15. of partitions: 50
          14.063911262269487
        16. of partitions: 60
          12.51514637879624
        17. of partitions: 70
          12.309312883565598
        18. of partitions: 80
          29.554991118293103
        19. of partitions: 90
          37.39714123885218
        20. of partitions: 100
          11.820253804288662
          ```

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/jihoonson/tajo-2 TAJO-1716

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/659.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #659


        commit f26cfd789fe0a8ce464049aae3b37278c6d72795
        Author: Jihoon Son <jihoonson@apache.org>
        Date: 2015-07-28T11:23:39Z

        TAJO-1716


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user jihoonson opened a pull request: https://github.com/apache/tajo/pull/659 TAJO-1716 : Repartitioner.makeEvenDistributedFetchImpl() does not distribute fetches evenly Here is the evaluation result. # of samples: 1000 # of fetches: 2000 Before patch ``` of partitions: 10 std dev: 7239.203108077597 of partitions: 20 std dev: 2507.946123823235 of partitions: 30 std dev: 1337.669359578645 of partitions: 40 std dev: 847.1299428068963 of partitions: 50 std dev: 590.6100520648041 of partitions: 60 std dev: 441.75456483838553 of partitions: 70 std dev: 340.5741442844647 of partitions: 80 std dev: 270.92110936581224 of partitions: 90 std dev: 221.5819175222533 of partitions: 100 std dev: 184.4234215060522 ``` After patch ``` of partitions: 10 13.61763562654033 of partitions: 20 21.6693331706813 of partitions: 30 15.134361197689614 of partitions: 40 13.04185569631777 of partitions: 50 14.063911262269487 of partitions: 60 12.51514637879624 of partitions: 70 12.309312883565598 of partitions: 80 29.554991118293103 of partitions: 90 37.39714123885218 of partitions: 100 11.820253804288662 ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/jihoonson/tajo-2 TAJO-1716 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/659.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #659 commit f26cfd789fe0a8ce464049aae3b37278c6d72795 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-07-28T11:23:39Z TAJO-1716

          People

          • Assignee:
            jihoonson Jihoon Son
            Reporter:
            jihoonson Jihoon Son
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development