Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-983

Worker should directly read Intermediate data stored in localhost rather than fetching

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: Data Shuffle
    • Labels:
      None

      Description

      Currently, worker always fetches all intermediate via Fetcher and than store them in local file system even though some intermediate data already are stored in local file system. It is inefficient and causes unnecessary I/O and extra storage occupation. We should improve it.

      1. TAJO-983.140820.0.patch.txt
        15 kB
        Mai Hai Thanh
      2. TAJO-983.140822.patch.txt
        20 kB
        Mai Hai Thanh
      3. TAJO-983.140825.1.patch.txt
        19 kB
        Mai Hai Thanh
      4. TAJO-983.140902.patch
        20 kB
        Mai Hai Thanh
      5. TAJO-983.140904.patch
        20 kB
        Mai Hai Thanh
      6. TAJO-983.140916.patch
        20 kB
        Hyunsik Choi
      7. TAJO-983.140918.patch
        39 kB
        Mai Hai Thanh
      8. TAJO-983.140922.patch
        41 kB
        Mai Hai Thanh
      9. TAJO-983.140923.patch
        40 kB
        Mai Hai Thanh
      10. TAJO-983.140924-Hyunsik.patch
        41 kB
        Hyunsik Choi

        Issue Links

          Activity

          Hide
          jihoonson Jihoon Son added a comment -

          +1 for this issue.

          Show
          jihoonson Jihoon Son added a comment - +1 for this issue.
          Hide
          mhthanh Mai Hai Thanh added a comment -

          Hi Hyunsik Choi,
          I submitted a patch to solve this issue. Please review it.

          Show
          mhthanh Mai Hai Thanh added a comment - Hi Hyunsik Choi , I submitted a patch to solve this issue. Please review it.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12663061/TAJO-983.140820.0.patch.txt
          against master revision ea5ce54.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 211 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-core tajo-yarn-pullserver.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/478//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/478//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/478//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/478//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12663061/TAJO-983.140820.0.patch.txt against master revision ea5ce54. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 211 new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-core tajo-yarn-pullserver. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/478//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/478//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/478//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/478//console This message is automatically generated.
          Hide
          hyunsik Hyunsik Choi added a comment -

          Hi Mai Hai Thanh,

          Your approach looks good to me. BTW, I have some better approach for it. I'm sorry for not mentioning it earlier.

          Please see Task::localizeFetchedData() method. This method makes FileFragments for input data from fetched files. The FileFragments will be ultimately used for a sequential scanner or merge scanner. I think that we can directly make FileFragment with a original local path without unnecessary copying the files stored in local file system.

          How do you think about that?

          Best regards,
          Hyunsik

          Show
          hyunsik Hyunsik Choi added a comment - Hi Mai Hai Thanh , Your approach looks good to me. BTW, I have some better approach for it. I'm sorry for not mentioning it earlier. Please see Task::localizeFetchedData() method. This method makes FileFragments for input data from fetched files. The FileFragments will be ultimately used for a sequential scanner or merge scanner. I think that we can directly make FileFragment with a original local path without unnecessary copying the files stored in local file system. How do you think about that? Best regards, Hyunsik
          Hide
          mhthanh Mai Hai Thanh added a comment -

          Thank Hyunsik Choi!
          I investigated your interesting approach. However, I think that the effects of the two approaches are the same. In case there are multiple file chunks (which is almost always the case with not-so-small data), we have to merge them into one file, which can be a file to be returned by Fetcher::get() or a FileFragment. To merge multiple chunks, copying data is unavoidable. In case there is only 1 file chunk to be fetched and this chunk does not represents a complete file, we also have to copy the data. In case there is only 1 file chunk to be fetched and this chunk represents a complete file (this case happens only with very small data), we should theoretically avoid copying and use the file directly. Nevertheless, we have to treat this file as an exceptional case in later processing code because it is not stored in the conventional default folder and with the conventional file name format. Beside, because chunk size is by default limited to be only 8 KB, the copying of data is not a problem. So, to keep the code clean for ease of maintenance and because of the low cost (also, rare case), I prefer to keep the current approach.

          Show
          mhthanh Mai Hai Thanh added a comment - Thank Hyunsik Choi ! I investigated your interesting approach. However, I think that the effects of the two approaches are the same. In case there are multiple file chunks (which is almost always the case with not-so-small data), we have to merge them into one file, which can be a file to be returned by Fetcher::get() or a FileFragment. To merge multiple chunks, copying data is unavoidable. In case there is only 1 file chunk to be fetched and this chunk does not represents a complete file, we also have to copy the data. In case there is only 1 file chunk to be fetched and this chunk represents a complete file (this case happens only with very small data), we should theoretically avoid copying and use the file directly. Nevertheless, we have to treat this file as an exceptional case in later processing code because it is not stored in the conventional default folder and with the conventional file name format. Beside, because chunk size is by default limited to be only 8 KB, the copying of data is not a problem. So, to keep the code clean for ease of maintenance and because of the low cost (also, rare case), I prefer to keep the current approach.
          Hide
          hyunsik Hyunsik Choi added a comment -

          Hi Mai,

          The main reason why a task reads directly reads intermediate data without copying is to reduce extra storage occupation and unnecessary I/O overheads. In practice, intermediate data can be usually from few bytes up to hundreds of mega bytes. Some of them in general are already stored in local disk. For large intermediate, I believe that non copy approach is worthy.

          Also, a task which consumes intermediate data uses MergeScanner which reads multiple file fragments iteratively. So, we don't need to merge input files into one file.

          In addition, please take a look at TAJO-992 for per-node shuffler. Before TAJO-992, in hash shuffle, each task makes a number of small files. So, Fetcher merged small chunks stored in a number of small partitions into one stream and pulls the stream. After TAJO-992, each node makes single intermediate data file for each partition. A task fetches corresponding partition files from a number of workers and read directly them without merging them. So, the cost of reading a number of small files became cheap.

          Thanks,
          Hyunsik

          Show
          hyunsik Hyunsik Choi added a comment - Hi Mai, The main reason why a task reads directly reads intermediate data without copying is to reduce extra storage occupation and unnecessary I/O overheads. In practice, intermediate data can be usually from few bytes up to hundreds of mega bytes. Some of them in general are already stored in local disk. For large intermediate, I believe that non copy approach is worthy. Also, a task which consumes intermediate data uses MergeScanner which reads multiple file fragments iteratively. So, we don't need to merge input files into one file. In addition, please take a look at TAJO-992 for per-node shuffler. Before TAJO-992 , in hash shuffle, each task makes a number of small files. So, Fetcher merged small chunks stored in a number of small partitions into one stream and pulls the stream. After TAJO-992 , each node makes single intermediate data file for each partition. A task fetches corresponding partition files from a number of workers and read directly them without merging them. So, the cost of reading a number of small files became cheap. Thanks, Hyunsik
          Hide
          mhthanh Mai Hai Thanh added a comment -

          Hi Hyunsik,
          I noticed the commit of TAJO-992 about 2 hours ago and already looked at it.
          If intermediate data can be up to hundreds of mega bytes, it is worth to implement a non copy approach. I will try it.

          Beside, I want to ask about the intermediate files. As I understand, an intermediate file can be stored in a remote host. So, Tajo's pull server uses Http protocol to transfer them from the remote host to the local host. Is it right ? (if this is right, then reading data in local host whenever possible has a benefit of avoiding Http file transfer. I guess file copy should be faster than Http file transfer even for local host's transfer. This is another benefit beside reducing I/O and extra storage)

          Show
          mhthanh Mai Hai Thanh added a comment - Hi Hyunsik, I noticed the commit of TAJO-992 about 2 hours ago and already looked at it. If intermediate data can be up to hundreds of mega bytes, it is worth to implement a non copy approach. I will try it. Beside, I want to ask about the intermediate files. As I understand, an intermediate file can be stored in a remote host. So, Tajo's pull server uses Http protocol to transfer them from the remote host to the local host. Is it right ? (if this is right, then reading data in local host whenever possible has a benefit of avoiding Http file transfer. I guess file copy should be faster than Http file transfer even for local host's transfer. This is another benefit beside reducing I/O and extra storage)
          Hide
          hyunsik Hyunsik Choi added a comment -

          Yes, you are right. Most of intermediate data are stored in remote hosts, and some data are stored in local disk. Theoretically, if some data of a query are evenly shuffled into N partitions (intermediate data) in the cluster where a number of cluster nodes N, 1/N data will be stored in the local disk.

          If tens of cluster nodes handle several tera bytes, we usually can see tens of giga bytes of aggregated intermediate data across the cluster nodes; some ETL jobs handles a large volume of intermediate data similar to that of the input data. Even though only some of them are stored in local disks, it would be better if we avoid unnecessary reads and writes.

          Best regards,
          Hyunsik

          Show
          hyunsik Hyunsik Choi added a comment - Yes, you are right. Most of intermediate data are stored in remote hosts, and some data are stored in local disk. Theoretically, if some data of a query are evenly shuffled into N partitions (intermediate data) in the cluster where a number of cluster nodes N, 1/N data will be stored in the local disk. If tens of cluster nodes handle several tera bytes, we usually can see tens of giga bytes of aggregated intermediate data across the cluster nodes; some ETL jobs handles a large volume of intermediate data similar to that of the input data. Even though only some of them are stored in local disks, it would be better if we avoid unnecessary reads and writes. Best regards, Hyunsik
          Hide
          mhthanh Mai Hai Thanh added a comment -

          Hi Hyunsik,
          I uploaded a new patch that implements the non-copy approach. Please see it

          Show
          mhthanh Mai Hai Thanh added a comment - Hi Hyunsik, I uploaded a new patch that implements the non-copy approach. Please see it
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12663624/TAJO-983.140822.patch.txt
          against master revision 839081a.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 214 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-core tajo-yarn-pullserver.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/479//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/479//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/479//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/479//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12663624/TAJO-983.140822.patch.txt against master revision 839081a. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 214 new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-core tajo-yarn-pullserver. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/479//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/479//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/479//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/479//console This message is automatically generated.
          Hide
          mhthanh Mai Hai Thanh added a comment -

          This new patch resolves some conflicts with recent commits from other issues.

          Hi Hyunsik Choi, may you check this latest patch ? This one already implements the non-copy approach based on your suggestion. All tests passed successfully. Since there are some other issues that deal with Task and TaskRunner (e.g., TAJO-855), it will be better if we can close this issue early to avoid conflicts with other issues.

          Show
          mhthanh Mai Hai Thanh added a comment - This new patch resolves some conflicts with recent commits from other issues. Hi Hyunsik Choi , may you check this latest patch ? This one already implements the non-copy approach based on your suggestion. All tests passed successfully. Since there are some other issues that deal with Task and TaskRunner (e.g., TAJO-855 ), it will be better if we can close this issue early to avoid conflicts with other issues.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12664072/TAJO-983.140825.patch.txt
          against master revision 7603a3d.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail.

          -1 release audit. The applied patch generated 101 release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-core tajo-yarn-pullserver.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/480//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/480//artifact/incubator-tajo/patchprocess/patchReleaseAuditProblems.txt
          Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/480//findbugsResult
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/480//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12664072/TAJO-983.140825.patch.txt against master revision 7603a3d. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail. -1 release audit. The applied patch generated 101 release audit warnings. +1 core tests. The patch passed unit tests in tajo-core tajo-yarn-pullserver. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/480//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/480//artifact/incubator-tajo/patchprocess/patchReleaseAuditProblems.txt Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/480//findbugsResult Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/480//console This message is automatically generated.
          Hide
          mhthanh Mai Hai Thanh added a comment -

          Fixed a minor hidden bug.

          Show
          mhthanh Mai Hai Thanh added a comment - Fixed a minor hidden bug.
          Hide
          hyunsik Hyunsik Choi added a comment -

          Thanks Mai! I'm reviewing this. I'll give comments soon.

          Show
          hyunsik Hyunsik Choi added a comment - Thanks Mai! I'm reviewing this. I'll give comments soon.
          Hide
          mhthanh Mai Hai Thanh added a comment -

          Hi Hyunsik Choi, how do you think about the patch ?

          Show
          mhthanh Mai Hai Thanh added a comment - Hi Hyunsik Choi , how do you think about the patch ?
          Hide
          hyunsik Hyunsik Choi added a comment -

          Hi Mai,

          I'm sorry for late response. Due to some urgent issues about my work, I couldn't finish the review. I'll comment by today's night.

          Thanks,
          Hyunsik

          Show
          hyunsik Hyunsik Choi added a comment - Hi Mai, I'm sorry for late response. Due to some urgent issues about my work, I couldn't finish the review. I'll comment by today's night. Thanks, Hyunsik
          Hide
          hyunsik Hyunsik Choi added a comment -

          Hi Mai,

          Thank you for your contribution. In overall, your patch looks good to me. The patch passes all unit tests.

          I tested the patch in a single local cluster. Hash shuffle worked correctly as follows:

          default> select count(*) from lineitem;
          Progress: 0%, response time: 0.193 sec
          Progress: 0%, response time: 0.194 sec
          Progress: 0%, response time: 0.395 sec
          Progress: 4%, response time: 0.796 sec
          Progress: 10%, response time: 1.398 sec
          Progress: 19%, response time: 2.2 sec
          Progress: 30%, response time: 3.202 sec
          Progress: 41%, response time: 4.205 sec
          Progress: 100%, response time: 4.994 sec
          ?count
          -------------------------------
          6001216
          (1 rows, 4.994 sec, 8 B selected)
          

          But, the patch causes file not found error when I tested range shuffle as follows:

          default> select l_orderkey, l_partkey from lineitem order by l_orderkey;
          Progress: 0%, response time: 0.622 sec
          Progress: 0%, response time: 0.623 sec
          Progress: 0%, response time: 0.825 sec
          Progress: 0%, response time: 1.239 sec
          Progress: 0%, response time: 1.841 sec
          Progress: 4%, response time: 2.642 sec
          Progress: 6%, response time: 3.644 sec
          Progress: 10%, response time: 4.646 sec
          Progress: 15%, response time: 5.647 sec
          Progress: 17%, response time: 6.65 sec
          Progress: 21%, response time: 7.652 sec
          Progress: 26%, response time: 8.654 sec
          Progress: 28%, response time: 9.656 sec
          Progress: 32%, response time: 10.678 sec
          Progress: 36%, response time: 11.68 sec
          Progress: 41%, response time: 12.681 sec
          Progress: 43%, response time: 13.684 sec
          Progress: 47%, response time: 14.687 sec
          Progress: 50%, response time: 15.69 sec
          Progress: 50%, response time: 16.692 sec
          Progress: 75%, response time: 17.694 sec
          Progress: 75%, response time: 18.696 sec
          ERROR: /tmp/tajo-hyunsik/tmpdir/q_1409368383100_0001/output/1/11_0/output/output (No such file or directory)
          java.io.FileNotFoundException: /tmp/tajo-hyunsik/tmpdir/q_1409368383100_0001/output/1/11_0/output/output (No such file or directory)
          	at java.io.FileInputStream.open(Native Method)
          	at java.io.FileInputStream.<init>(FileInputStream.java:146)
          	at org.apache.tajo.storage.RawFile$RawFileScanner.init(RawFile.java:85)
          	at org.apache.tajo.engine.planner.physical.ExternalSortExec$PairWiseMerger.init(ExternalSortExec.java:618)
          	at org.apache.tajo.engine.planner.physical.ExternalSortExec$PairWiseMerger.init(ExternalSortExec.java:618)
          	at org.apache.tajo.engine.planner.physical.ExternalSortExec$PairWiseMerger.init(ExternalSortExec.java:617)
          	at org.apache.tajo.engine.planner.physical.ExternalSortExec.next(ExternalSortExec.java:299)
          	at org.apache.tajo.engine.planner.physical.StoreTableExec.next(StoreTableExec.java:112)
          	at org.apache.tajo.worker.Task.run(Task.java:454)
          	at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:445)
          	at java.lang.Thread.run(Thread.java:744)
          

          Also, you copy some code to build local file paths from TajoPullServerService::messageReceived. It would be great if you refactor the part into some methods and reuse the methods.

          In addition, it will be easier for reviewers to leave comments if you upload your patch to reviewboard (https://reviews.apache.org/dashboard/) or github pull request (https://github.com/apache/tajo/pulls).

          Thanks!

          Show
          hyunsik Hyunsik Choi added a comment - Hi Mai, Thank you for your contribution. In overall, your patch looks good to me. The patch passes all unit tests. I tested the patch in a single local cluster. Hash shuffle worked correctly as follows: default > select count(*) from lineitem; Progress: 0%, response time: 0.193 sec Progress: 0%, response time: 0.194 sec Progress: 0%, response time: 0.395 sec Progress: 4%, response time: 0.796 sec Progress: 10%, response time: 1.398 sec Progress: 19%, response time: 2.2 sec Progress: 30%, response time: 3.202 sec Progress: 41%, response time: 4.205 sec Progress: 100%, response time: 4.994 sec ?count ------------------------------- 6001216 (1 rows, 4.994 sec, 8 B selected) But, the patch causes file not found error when I tested range shuffle as follows: default> select l_orderkey, l_partkey from lineitem order by l_orderkey; Progress: 0%, response time: 0.622 sec Progress: 0%, response time: 0.623 sec Progress: 0%, response time: 0.825 sec Progress: 0%, response time: 1.239 sec Progress: 0%, response time: 1.841 sec Progress: 4%, response time: 2.642 sec Progress: 6%, response time: 3.644 sec Progress: 10%, response time: 4.646 sec Progress: 15%, response time: 5.647 sec Progress: 17%, response time: 6.65 sec Progress: 21%, response time: 7.652 sec Progress: 26%, response time: 8.654 sec Progress: 28%, response time: 9.656 sec Progress: 32%, response time: 10.678 sec Progress: 36%, response time: 11.68 sec Progress: 41%, response time: 12.681 sec Progress: 43%, response time: 13.684 sec Progress: 47%, response time: 14.687 sec Progress: 50%, response time: 15.69 sec Progress: 50%, response time: 16.692 sec Progress: 75%, response time: 17.694 sec Progress: 75%, response time: 18.696 sec ERROR: /tmp/tajo-hyunsik/tmpdir/q_1409368383100_0001/output/1/11_0/output/output (No such file or directory) java.io.FileNotFoundException: /tmp/tajo-hyunsik/tmpdir/q_1409368383100_0001/output/1/11_0/output/output (No such file or directory) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:146) at org.apache.tajo.storage.RawFile$RawFileScanner.init(RawFile.java:85) at org.apache.tajo.engine.planner.physical.ExternalSortExec$PairWiseMerger.init(ExternalSortExec.java:618) at org.apache.tajo.engine.planner.physical.ExternalSortExec$PairWiseMerger.init(ExternalSortExec.java:618) at org.apache.tajo.engine.planner.physical.ExternalSortExec$PairWiseMerger.init(ExternalSortExec.java:617) at org.apache.tajo.engine.planner.physical.ExternalSortExec.next(ExternalSortExec.java:299) at org.apache.tajo.engine.planner.physical.StoreTableExec.next(StoreTableExec.java:112) at org.apache.tajo.worker.Task.run(Task.java:454) at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:445) at java.lang.Thread.run(Thread.java:744) Also, you copy some code to build local file paths from TajoPullServerService::messageReceived. It would be great if you refactor the part into some methods and reuse the methods. In addition, it will be easier for reviewers to leave comments if you upload your patch to reviewboard ( https://reviews.apache.org/dashboard/ ) or github pull request ( https://github.com/apache/tajo/pulls ). Thanks!
          Hide
          mhthanh Mai Hai Thanh added a comment - - edited

          Thank Hyunsik! I fixed the bug and uploaded the new patch to both this Jira page and the review board: https://reviews.apache.org/r/25242/

          I also tried to refactor the code to avoid code copying. However, after refactoring, many tests failed due to an exception throw by Netty and I could not find a solution. In summary, except the copying of some code, other things go well.

          Show
          mhthanh Mai Hai Thanh added a comment - - edited Thank Hyunsik! I fixed the bug and uploaded the new patch to both this Jira page and the review board: https://reviews.apache.org/r/25242/ I also tried to refactor the code to avoid code copying. However, after refactoring, many tests failed due to an exception throw by Netty and I could not find a solution. In summary, except the copying of some code, other things go well.
          Hide
          mhthanh Mai Hai Thanh added a comment - - edited

          Hi Hyunsik,
          I uploaded a new patch file to resolve some (minor) conflicts with TAJO-991. As I mentioned in the previous comment, beside uploading the file in this website, I also uploaded the new patch file to the review board: https://reviews.apache.org/r/25242

          Show
          mhthanh Mai Hai Thanh added a comment - - edited Hi Hyunsik, I uploaded a new patch file to resolve some (minor) conflicts with TAJO-991 . As I mentioned in the previous comment, beside uploading the file in this website, I also uploaded the new patch file to the review board: https://reviews.apache.org/r/25242
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12666435/TAJO-983.140904.patch
          against master revision 0e43627.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 222 new Findbugs (version 2.0.3) warnings.

          -1 release audit. The applied patch generated 101 release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-core tajo-yarn-pullserver.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/484//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/484//artifact/incubator-tajo/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/484//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/484//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/484//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12666435/TAJO-983.140904.patch against master revision 0e43627. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 222 new Findbugs (version 2.0.3) warnings. -1 release audit. The applied patch generated 101 release audit warnings. +1 core tests. The patch passed unit tests in tajo-core tajo-yarn-pullserver. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/484//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/484//artifact/incubator-tajo/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/484//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/484//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/484//console This message is automatically generated.
          Hide
          hyunsik Hyunsik Choi added a comment -

          I updated the rebased patch against the latest revision.

          Show
          hyunsik Hyunsik Choi added a comment - I updated the rebased patch against the latest revision.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12668940/TAJO-983.140916.patch
          against master revision 15450e8.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail.

          -1 release audit. The applied patch generated 101 release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-core tajo-yarn-pullserver.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/490//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/490//artifact/incubator-tajo/patchprocess/patchReleaseAuditProblems.txt
          Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/490//findbugsResult
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/490//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12668940/TAJO-983.140916.patch against master revision 15450e8. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail. -1 release audit. The applied patch generated 101 release audit warnings. +1 core tests. The patch passed unit tests in tajo-core tajo-yarn-pullserver. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/490//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/490//artifact/incubator-tajo/patchprocess/patchReleaseAuditProblems.txt Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/490//findbugsResult Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/490//console This message is automatically generated.
          Hide
          mhthanh Mai Hai Thanh added a comment -

          I update the patch to solve FileNotFound Exception which is caused by the deletion of intermediate data.

          Show
          mhthanh Mai Hai Thanh added a comment - I update the patch to solve FileNotFound Exception which is caused by the deletion of intermediate data.
          Hide
          mhthanh Mai Hai Thanh added a comment -

          I update the patch to solve FileNotFound Exception which is caused by the deletion of intermediate data. (some redundant white spaces in the patch file is deleted and the file is uploaded again)

          Show
          mhthanh Mai Hai Thanh added a comment - I update the patch to solve FileNotFound Exception which is caused by the deletion of intermediate data. (some redundant white spaces in the patch file is deleted and the file is uploaded again)
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12669654/TAJO-983.140918.patch
          against master revision 2001774.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 220 new Findbugs (version 2.0.3) warnings.

          -1 release audit. The applied patch generated 105 release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-core tajo-storage tajo-yarn-pullserver.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/495//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/495//artifact/incubator-tajo/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/495//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/495//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/495//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-storage.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/495//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12669654/TAJO-983.140918.patch against master revision 2001774. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 220 new Findbugs (version 2.0.3) warnings. -1 release audit. The applied patch generated 105 release audit warnings. +1 core tests. The patch passed unit tests in tajo-core tajo-storage tajo-yarn-pullserver. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/495//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/495//artifact/incubator-tajo/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/495//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/495//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/495//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-storage.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/495//console This message is automatically generated.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12669661/TAJO-983.140918.patch
          against master revision 2001774.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 220 new Findbugs (version 2.0.3) warnings.

          -1 release audit. The applied patch generated 104 release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-core tajo-storage tajo-yarn-pullserver.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/496//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/496//artifact/incubator-tajo/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/496//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/496//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/496//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-storage.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/496//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12669661/TAJO-983.140918.patch against master revision 2001774. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 220 new Findbugs (version 2.0.3) warnings. -1 release audit. The applied patch generated 104 release audit warnings. +1 core tests. The patch passed unit tests in tajo-core tajo-storage tajo-yarn-pullserver. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/496//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/496//artifact/incubator-tajo/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/496//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/496//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/496//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-storage.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/496//console This message is automatically generated.
          Hide
          mhthanh Mai Hai Thanh added a comment -

          I update the patch to incorporate Hyunsik's suggestion in the review board.

          Show
          mhthanh Mai Hai Thanh added a comment - I update the patch to incorporate Hyunsik's suggestion in the review board.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12670352/TAJO-983.140922.patch
          against master revision 7510f88.

          -1 patch. The patch command could not apply the patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/505//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12670352/TAJO-983.140922.patch against master revision 7510f88. -1 patch. The patch command could not apply the patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/505//console This message is automatically generated.
          Hide
          mhthanh Mai Hai Thanh added a comment -

          Hi Hyunsik,
          I guess that TajoQA failed to apply the latest patch because of possible conflicts with some commits in the last few days. However, because I modified on master branch (sorry for my naive mistake and inexperience), I don't know how to rebase it to the latest Tajo code base. If you think that the patch is ready to commit and know how to rebase it quickly, please help me to rebase it to the master branch as you did once before. Otherwise, in the worst case, I will checkout Tajo's latest code and make the changes manually from scratch.

          Show
          mhthanh Mai Hai Thanh added a comment - Hi Hyunsik, I guess that TajoQA failed to apply the latest patch because of possible conflicts with some commits in the last few days. However, because I modified on master branch (sorry for my naive mistake and inexperience), I don't know how to rebase it to the latest Tajo code base. If you think that the patch is ready to commit and know how to rebase it quickly, please help me to rebase it to the master branch as you did once before. Otherwise, in the worst case, I will checkout Tajo's latest code and make the changes manually from scratch.
          Hide
          hyunsik Hyunsik Choi added a comment - - edited

          Hi Mai,

          If you work on source code which is checked out via git, just execute the following command:

          git pull origin master
          

          pull command fetch and merge the latest updated revision to your working revision. Before executing git pull, you need to commit your working source.

          Show
          hyunsik Hyunsik Choi added a comment - - edited Hi Mai, If you work on source code which is checked out via git, just execute the following command: git pull origin master pull command fetch and merge the latest updated revision to your working revision. Before executing git pull , you need to commit your working source.
          Hide
          mhthanh Mai Hai Thanh added a comment -

          Hi Hyunsik, I updated the patch with the latest Tajo code base

          Show
          mhthanh Mai Hai Thanh added a comment - Hi Hyunsik, I updated the patch with the latest Tajo code base
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12670577/TAJO-983.140923.patch
          against master revision 7510f88.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 222 new Findbugs (version 2.0.3) warnings.

          -1 release audit. The applied patch generated 105 release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-core tajo-storage tajo-yarn-pullserver.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/507//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/507//artifact/incubator-tajo/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/507//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/507//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/507//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-storage.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/507//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12670577/TAJO-983.140923.patch against master revision 7510f88. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 222 new Findbugs (version 2.0.3) warnings. -1 release audit. The applied patch generated 105 release audit warnings. +1 core tests. The patch passed unit tests in tajo-core tajo-storage tajo-yarn-pullserver. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/507//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/507//artifact/incubator-tajo/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/507//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/507//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/507//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-storage.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/507//console This message is automatically generated.
          Hide
          hyunsik Hyunsik Choi added a comment -

          +1 for your latest patch.

          I've touched indent and some trivial things on your patch:

          • Removed useLocalFile flag from Fetcher because FileChunk::fromRemote is logically equivalent to useLocalFile flag.
          • In 734 line of Task, I added some code to skip creation of storeChunk when storeChunk retrieved from getLocalStoredFileChunk is null. It's because storeChunk will be NULL and this case is normal state when a range request is out of range.
          • Replaced "interFile_" string used in ExternalSortExec by some defined constant variable. I also added the prefix character '@' - which can not be used for table name identifier - in order to avoid potential possibility of duplicated fragment names.

          If you agree with my additional change, I'll commit the patch including my change.

          Thank you very much for your contribution!

          Show
          hyunsik Hyunsik Choi added a comment - +1 for your latest patch. I've touched indent and some trivial things on your patch: Removed useLocalFile flag from Fetcher because FileChunk::fromRemote is logically equivalent to useLocalFile flag. In 734 line of Task, I added some code to skip creation of storeChunk when storeChunk retrieved from getLocalStoredFileChunk is null. It's because storeChunk will be NULL and this case is normal state when a range request is out of range. Replaced "interFile_" string used in ExternalSortExec by some defined constant variable. I also added the prefix character '@' - which can not be used for table name identifier - in order to avoid potential possibility of duplicated fragment names. If you agree with my additional change, I'll commit the patch including my change. Thank you very much for your contribution!
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12670823/TAJO-983.140924-Hyunsik.patch
          against master revision 5a7e272.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 222 new Findbugs (version 2.0.3) warnings.

          -1 release audit. The applied patch generated 104 release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-core tajo-storage tajo-yarn-pullserver.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/511//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/511//artifact/incubator-tajo/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/511//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/511//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/511//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-storage.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/511//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12670823/TAJO-983.140924-Hyunsik.patch against master revision 5a7e272. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 222 new Findbugs (version 2.0.3) warnings. -1 release audit. The applied patch generated 104 release audit warnings. +1 core tests. The patch passed unit tests in tajo-core tajo-storage tajo-yarn-pullserver. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/511//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/511//artifact/incubator-tajo/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/511//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/511//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/511//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-storage.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/511//console This message is automatically generated.
          Hide
          mhthanh Mai Hai Thanh added a comment -

          Thank Hyunsik,
          I agree with your additional changes, which clearly make the patch better. Please go on! It's my pleasure to make contributions to Tajo.

          Beside, I have also read your notice on the review board about the coding standard in which I should use 2 spaces instead of tab. I am sorry about that. To use spaces instead of tabs, actually, I have already checked the check-box "Insert spaces for tabs" (default is un-checked, in menu Window -> Preferences -> General -> Editors -> Text Editors), and then use Shift+Ctrl+F to format the code automatically. So, I thought that the code had already followed the coding standard and I don't know what I should do further. When you have time, please make a short instruction in the document about this. I am sure that it will be useful for new comers, especially the ones who do not have much experience with Eclipse like me!

          Show
          mhthanh Mai Hai Thanh added a comment - Thank Hyunsik, I agree with your additional changes, which clearly make the patch better. Please go on! It's my pleasure to make contributions to Tajo. Beside, I have also read your notice on the review board about the coding standard in which I should use 2 spaces instead of tab. I am sorry about that. To use spaces instead of tabs, actually, I have already checked the check-box "Insert spaces for tabs" (default is un-checked, in menu Window -> Preferences -> General -> Editors -> Text Editors), and then use Shift+Ctrl+F to format the code automatically. So, I thought that the code had already followed the coding standard and I don't know what I should do further. When you have time, please make a short instruction in the document about this. I am sure that it will be useful for new comers, especially the ones who do not have much experience with Eclipse like me!
          Hide
          hyunsik Hyunsik Choi added a comment -

          +1
          This work makes shuffle more stable and reduces unnecessary I/O and network traffic during shuffle. I greatly appreciate your nice contributions. I'll commit it shortly.

          Also, I'll give comment about coding standards tomorrow. Here is too late

          Show
          hyunsik Hyunsik Choi added a comment - +1 This work makes shuffle more stable and reduces unnecessary I/O and network traffic during shuffle. I greatly appreciate your nice contributions. I'll commit it shortly. Also, I'll give comment about coding standards tomorrow. Here is too late
          Hide
          hyunsik Hyunsik Choi added a comment -

          I've just committed the latest patch to master branch. Fixing it as resolved.

          Show
          hyunsik Hyunsik Choi added a comment - I've just committed the latest patch to master branch. Fixing it as resolved.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #374 (See https://builds.apache.org/job/Tajo-master-build/374/)
          TAJO-983: Worker should directly read Intermediate data stored in localhost rather than fetching. (Mai Hai Thanh via hyunsik) (hyunsik: rev 644b7cd991ea402115c6dc1d198e7f1d7f41771b)

          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/UniformRangePartition.java
          • tajo-core/src/main/java/org/apache/tajo/worker/Task.java
          • tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java
          • tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java
          • tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java
          • tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java
          • CHANGES
          • tajo-storage/src/main/java/org/apache/tajo/storage/RawFile.java
          • tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/PullServerAuxService.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #374 (See https://builds.apache.org/job/Tajo-master-build/374/ ) TAJO-983 : Worker should directly read Intermediate data stored in localhost rather than fetching. (Mai Hai Thanh via hyunsik) (hyunsik: rev 644b7cd991ea402115c6dc1d198e7f1d7f41771b) tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java tajo-core/src/main/java/org/apache/tajo/engine/planner/UniformRangePartition.java tajo-core/src/main/java/org/apache/tajo/worker/Task.java tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java CHANGES tajo-storage/src/main/java/org/apache/tajo/storage/RawFile.java tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/PullServerAuxService.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-CODEGEN-build #16 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/16/)
          TAJO-983: Worker should directly read Intermediate data stored in localhost rather than fetching. (Mai Hai Thanh via hyunsik) (hyunsik: rev 644b7cd991ea402115c6dc1d198e7f1d7f41771b)

          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java
          • CHANGES
          • tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/RawFile.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/UniformRangePartition.java
          • tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java
          • tajo-core/src/main/java/org/apache/tajo/worker/Task.java
          • tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java
          • tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/PullServerAuxService.java
          • tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-CODEGEN-build #16 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/16/ ) TAJO-983 : Worker should directly read Intermediate data stored in localhost rather than fetching. (Mai Hai Thanh via hyunsik) (hyunsik: rev 644b7cd991ea402115c6dc1d198e7f1d7f41771b) tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java CHANGES tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java tajo-storage/src/main/java/org/apache/tajo/storage/RawFile.java tajo-core/src/main/java/org/apache/tajo/engine/planner/UniformRangePartition.java tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java tajo-core/src/main/java/org/apache/tajo/worker/Task.java tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/PullServerAuxService.java tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-block_iteration-branch-build #5 (See https://builds.apache.org/job/Tajo-block_iteration-branch-build/5/)
          TAJO-983: Worker should directly read Intermediate data stored in localhost rather than fetching. (Mai Hai Thanh via hyunsik) (hyunsik: rev 644b7cd991ea402115c6dc1d198e7f1d7f41771b)

          • tajo-core/src/main/java/org/apache/tajo/worker/Task.java
          • tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/PullServerAuxService.java
          • CHANGES
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/UniformRangePartition.java
          • tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java
          • tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/RawFile.java
          • tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java
          • tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-block_iteration-branch-build #5 (See https://builds.apache.org/job/Tajo-block_iteration-branch-build/5/ ) TAJO-983 : Worker should directly read Intermediate data stored in localhost rather than fetching. (Mai Hai Thanh via hyunsik) (hyunsik: rev 644b7cd991ea402115c6dc1d198e7f1d7f41771b) tajo-core/src/main/java/org/apache/tajo/worker/Task.java tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/PullServerAuxService.java CHANGES tajo-core/src/main/java/org/apache/tajo/engine/planner/UniformRangePartition.java tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java tajo-storage/src/main/java/org/apache/tajo/storage/RawFile.java tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java

            People

            • Assignee:
              mhthanh Mai Hai Thanh
              Reporter:
              hyunsik Hyunsik Choi
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development