Details
-
Type:
Improvement
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 0.9.0
-
Component/s: Data Shuffle
-
Labels:None
Description
Currently, shuffle uri use the string field. but most params is a number in uri
We need change to Varint of protocol buffer.
https://developers.google.com/protocol-buffers/docs/encoding#varints
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Created a review request against branch master in reviewboard
https://reviews.apache.org/r/20763/
Thank you for your detailed explains.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12642176/TAJO-789.patch
against master revision 71f394d.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The applied patch does not increase the total number of javadoc warnings.
+1 checkstyle. The patch generated 0 code style errors.
-1 findbugs. The patch appears to introduce 189 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in tajo-core.
Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/389//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/389//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/389//console
This message is automatically generated.
Updated the review request against branch master in reviewboard
https://reviews.apache.org/r/20763/
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12642189/TAJO-789_20140428_15%3A22%3A57.patch
against master revision 71f394d.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The applied patch does not increase the total number of javadoc warnings.
+1 checkstyle. The patch generated 0 code style errors.
-1 findbugs. The patch appears to introduce 189 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in tajo-core.
Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/390//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/390//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/390//console
This message is automatically generated.
This issue got +1 from reviewboard.
I've just committed
SUCCESS: Integrated in Tajo-master-build #205 (See https://builds.apache.org/job/Tajo-master-build/205/)
In addition, I explain the issue in more detail. Currently, QueryMaster collects all shuffled data set written by all tasks, and then QueryMaster directly sends URIs to each worker. Each worker pulls the shuffled data which resides in each node via URI. BTW, URI is usually is long. It may consume lots of network bandwidth. The main objective of this issue is to reduce the information to indicate the locations of intermediate data.