Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: Data Shuffle
    • Labels:
      None

      Description

      Currently, shuffle uri use the string field. but most params is a number in uri
      We need change to Varint of protocol buffer.
      https://developers.google.com/protocol-buffers/docs/encoding#varints

      1. TAJO-789.patch
        51 kB
        Jinho Kim
      2. TAJO-789_20140428_15:22:57.patch
        52 kB
        Jinho Kim

        Activity

        Hide
        hyunsik Hyunsik Choi added a comment -

        In addition, I explain the issue in more detail. Currently, QueryMaster collects all shuffled data set written by all tasks, and then QueryMaster directly sends URIs to each worker. Each worker pulls the shuffled data which resides in each node via URI. BTW, URI is usually is long. It may consume lots of network bandwidth. The main objective of this issue is to reduce the information to indicate the locations of intermediate data.

        Show
        hyunsik Hyunsik Choi added a comment - In addition, I explain the issue in more detail. Currently, QueryMaster collects all shuffled data set written by all tasks, and then QueryMaster directly sends URIs to each worker. Each worker pulls the shuffled data which resides in each node via URI. BTW, URI is usually is long. It may consume lots of network bandwidth. The main objective of this issue is to reduce the information to indicate the locations of intermediate data.
        Hide
        jhkim Jinho Kim added a comment -

        Created a review request against branch master in reviewboard
        https://reviews.apache.org/r/20763/

        Show
        jhkim Jinho Kim added a comment - Created a review request against branch master in reviewboard https://reviews.apache.org/r/20763/
        Hide
        jhkim Jinho Kim added a comment -

        Thank you for your detailed explains.

        Show
        jhkim Jinho Kim added a comment - Thank you for your detailed explains.
        Hide
        tajoqa Tajo QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12642176/TAJO-789.patch
        against master revision 71f394d.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

        +1 checkstyle. The patch generated 0 code style errors.

        -1 findbugs. The patch appears to introduce 189 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in tajo-core.

        Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/389//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/389//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
        Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/389//console

        This message is automatically generated.

        Show
        tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642176/TAJO-789.patch against master revision 71f394d. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 189 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-core. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/389//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/389//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/389//console This message is automatically generated.
        Hide
        jhkim Jinho Kim added a comment -

        Updated the review request against branch master in reviewboard
        https://reviews.apache.org/r/20763/

        Show
        jhkim Jinho Kim added a comment - Updated the review request against branch master in reviewboard https://reviews.apache.org/r/20763/
        Hide
        tajoqa Tajo QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12642189/TAJO-789_20140428_15%3A22%3A57.patch
        against master revision 71f394d.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

        +1 checkstyle. The patch generated 0 code style errors.

        -1 findbugs. The patch appears to introduce 189 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in tajo-core.

        Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/390//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/390//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
        Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/390//console

        This message is automatically generated.

        Show
        tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642189/TAJO-789_20140428_15%3A22%3A57.patch against master revision 71f394d. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 189 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-core. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/390//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/390//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/390//console This message is automatically generated.
        Hide
        jhkim Jinho Kim added a comment -

        This issue got +1 from reviewboard.
        I've just committed

        Show
        jhkim Jinho Kim added a comment - This issue got +1 from reviewboard. I've just committed
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #205 (See https://builds.apache.org/job/Tajo-master-build/205/)

        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #205 (See https://builds.apache.org/job/Tajo-master-build/205/ )

          People

          • Assignee:
            jhkim Jinho Kim
            Reporter:
            jhkim Jinho Kim
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development