Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: build
    • Labels:

      Description

      Hadoop 2.2.0 has been released. We need to bump up hadoop to 2.2.0. This hadoop version uses protobuf-2.5.0. So, Tajo also needs to bump up protobuf-2.5.0.

      1. TAJO-135.patch
        75 kB
        Jihoon Son
      2. TAJO-135_3.patch
        74 kB
        Jihoon Son
      3. TAJO-135_2.patch
        75 kB
        Jihoon Son

        Issue Links

          Activity

          Hide
          Jihoon Son added a comment -

          I'll start this issue.

          Show
          Jihoon Son added a comment - I'll start this issue.
          Hide
          Jihoon Son added a comment -

          Since Hive uses protobuf-2.4.1, I cannot proceed this issue until the Hive dependency is separated from the tajo-core dependency.

          Show
          Jihoon Son added a comment - Since Hive uses protobuf-2.4.1, I cannot proceed this issue until the Hive dependency is separated from the tajo-core dependency.
          Hide
          Jihoon Son added a comment -

          This patch passed 'mvn clean verify' successfully.

          Show
          Jihoon Son added a comment - This patch passed 'mvn clean verify' successfully.
          Hide
          Jihoon Son added a comment -

          I uploaded the second patch after removing some commented out codes.

          Show
          Jihoon Son added a comment - I uploaded the second patch after removing some commented out codes.
          Hide
          Jaehwa Jung added a comment -

          I failed at common module unit tests as follows:

          Results :
          
          Tests in error: 
            testCreate(org.apache.tajo.datum.protobuf.TestProtobufDatum): This is supposed to be overridden by subclasses.
            testWriteLoadProtoFromFile(org.apache.tajo.util.TestFileUtils): This is supposed to be overridden by subclasses.
            testWriteLoadProtoFromStream(org.apache.tajo.util.TestFileUtils): This is supposed to be overridden by subclasses.
            testWriteLoadProtoFromPath(org.apache.tajo.util.TestFileUtils): This is supposed to be overridden by subclasses.
          
          Tests run: 132, Failures: 0, Errors: 4, Skipped: 0
          
          Show
          Jaehwa Jung added a comment - I failed at common module unit tests as follows: Results : Tests in error: testCreate(org.apache.tajo.datum.protobuf.TestProtobufDatum): This is supposed to be overridden by subclasses. testWriteLoadProtoFromFile(org.apache.tajo.util.TestFileUtils): This is supposed to be overridden by subclasses. testWriteLoadProtoFromStream(org.apache.tajo.util.TestFileUtils): This is supposed to be overridden by subclasses. testWriteLoadProtoFromPath(org.apache.tajo.util.TestFileUtils): This is supposed to be overridden by subclasses. Tests run: 132, Failures: 0, Errors: 4, Skipped: 0
          Hide
          Jihoon Son added a comment -

          Thanks, JaeHwa.
          The above message means that the protobuf version is mismatched.
          Did you test with protobuf 2.5.0?

          Show
          Jihoon Son added a comment - Thanks, JaeHwa. The above message means that the protobuf version is mismatched. Did you test with protobuf 2.5.0?
          Hide
          Hyunsik Choi added a comment -

          JaeHwa,

          The error may be caused by old protoc version. Since hadoop-2.2.0, we should use protoc 2.5.0. I've verified 'mvn clean install' with protoc 2.5.0.

          Show
          Hyunsik Choi added a comment - JaeHwa, The error may be caused by old protoc version. Since hadoop-2.2.0, we should use protoc 2.5.0. I've verified 'mvn clean install' with protoc 2.5.0.
          Hide
          Jaehwa Jung added a comment -

          Sorry, Jihoon.

          I tested with protobuf 2.4.1.
          I'll try test again with protobuf 2.5.0.

          Show
          Jaehwa Jung added a comment - Sorry, Jihoon. I tested with protobuf 2.4.1. I'll try test again with protobuf 2.5.0.
          Hide
          Henry Saputra added a comment -

          HI Jihoon, since you modify a lot of the files would you mind updating the default comment from IntelliJ Idea "//To change body of implemented methods use File | Settings | File Templates." to something more useful. Thanks.

          Show
          Henry Saputra added a comment - HI Jihoon, since you modify a lot of the files would you mind updating the default comment from IntelliJ Idea "//To change body of implemented methods use File | Settings | File Templates." to something more useful. Thanks.
          Hide
          Jihoon Son added a comment -

          Thanks, Henry.
          Those comments are not added in this patch, but have been existed before this issue.
          In my opinion, since this issue just focuses on bumping up the hadoop version, it would be better that those comments are changed to something more useful in other issues.

          Show
          Jihoon Son added a comment - Thanks, Henry. Those comments are not added in this patch, but have been existed before this issue. In my opinion, since this issue just focuses on bumping up the hadoop version, it would be better that those comments are changed to something more useful in other issues.
          Hide
          Jihoon Son added a comment -

          Please review the patch.

          Show
          Jihoon Son added a comment - Please review the patch.
          Hide
          Hyunsik Choi added a comment -

          I'm sorry for the review. I'm reviewing the patch.

          Show
          Hyunsik Choi added a comment - I'm sorry for the review. I'm reviewing the patch.
          Hide
          Hyunsik Choi added a comment -

          The patch looks good for me. Some classes include unused imports caused by changes. It would be better if you remove them.

          In addition, I only tested it on a local cluster. After I test it on a real cluster, I'll give +1 for this patch. Thanks!

          Show
          Hyunsik Choi added a comment - The patch looks good for me. Some classes include unused imports caused by changes. It would be better if you remove them. In addition, I only tested it on a local cluster. After I test it on a real cluster, I'll give +1 for this patch. Thanks!
          Hide
          Jihoon Son added a comment -

          Thanks, Hyunsik.
          I uploaded the third patch after clearing unused imports.

          Show
          Jihoon Son added a comment - Thanks, Hyunsik. I uploaded the third patch after clearing unused imports.
          Hide
          Jinho Kim added a comment -

          ++1
          Great Jihoon!!
          I've tested the patch on real cluster. it worked well

          • De/Compression
          • TPC-H 1, 3
          Show
          Jinho Kim added a comment - ++1 Great Jihoon!! I've tested the patch on real cluster. it worked well De/Compression TPC-H 1, 3
          Hide
          Hyunsik Choi added a comment -

          +1
          I also verified the patch.

          Show
          Hyunsik Choi added a comment - +1 I also verified the patch.
          Hide
          Jihoon Son added a comment -

          Thanks guys.
          I've just committed the patch.

          Show
          Jihoon Son added a comment - Thanks guys. I've just committed the patch.
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Tajo-trunk-postcommit #637 (See https://builds.apache.org/job/Tajo-trunk-postcommit/637/)
          TAJO-135: Bump up hadoop to 2.2.0. (jihoon) (jihoonson: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=ecc749e811b9f978c9d9bb2cd9bfee6f795653df)

          • tajo-core/tajo-core-backend/src/main/proto/InternalTypes.proto
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/YarnContainerProxy.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/YarnTajoResourceManager.java
          • tajo-core/tajo-core-backend/src/main/proto/TajoMasterProtocol.proto
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorker.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMaster.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/AbstractTaskScheduler.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoResourceAllocator.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
          • tajo-core/tajo-core-backend/pom.xml
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterManagerService.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/GlobalEngine.java
          • tajo-common/src/main/proto/yarn_protos.proto
          • tajo-core/tajo-core-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java
          • tajo-core/tajo-core-pullserver/src/main/java/org/apache/tajo/pullserver/PullServerAuxService.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TaskRunnerManager.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TaskRunner.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryInProgress.java
          • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TestTajoIds.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/AbstractResourceAllocator.java
          • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TajoTestingCluster.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorkerManagerService.java
          • BUILDING.txt
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryJobManager.java
          • tajo-common/src/main/proto/Security.proto
          • tajo-core/tajo-core-pullserver/src/main/java/org/apache/tajo/pullserver/FadvisedFileRegion.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterRunner.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/YarnResourceAllocator.java
          • CHANGES.txt
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterTask.java
          • tajo-project/pom.xml
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMasterClientService.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoAsyncDispatcher.java
          • tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto
          • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/MiniTajoYarnCluster.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/YarnTaskRunnerLauncherImpl.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainer.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/TajoWorkerResourceManager.java
          • tajo-core/tajo-core-pullserver/src/main/java/org/apache/tajo/pullserver/FadvisedChunkedFile.java
          • tajo-core/tajo-core-backend/src/main/proto/TajoWorkerProtocol.proto
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/YarnRMContainerAllocator.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorkerClientService.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainerId.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/util/ApplicationIdUtils.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMaster.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMasterService.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/Query.java
          Show
          Hudson added a comment - FAILURE: Integrated in Tajo-trunk-postcommit #637 (See https://builds.apache.org/job/Tajo-trunk-postcommit/637/ ) TAJO-135 : Bump up hadoop to 2.2.0. (jihoon) (jihoonson: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=ecc749e811b9f978c9d9bb2cd9bfee6f795653df ) tajo-core/tajo-core-backend/src/main/proto/InternalTypes.proto tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/YarnContainerProxy.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/YarnTajoResourceManager.java tajo-core/tajo-core-backend/src/main/proto/TajoMasterProtocol.proto tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorker.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMaster.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/AbstractTaskScheduler.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoResourceAllocator.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java tajo-core/tajo-core-backend/pom.xml tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterManagerService.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/GlobalEngine.java tajo-common/src/main/proto/yarn_protos.proto tajo-core/tajo-core-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java tajo-core/tajo-core-pullserver/src/main/java/org/apache/tajo/pullserver/PullServerAuxService.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TaskRunnerManager.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TaskRunner.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryInProgress.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TestTajoIds.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/AbstractResourceAllocator.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TajoTestingCluster.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorkerManagerService.java BUILDING.txt tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryJobManager.java tajo-common/src/main/proto/Security.proto tajo-core/tajo-core-pullserver/src/main/java/org/apache/tajo/pullserver/FadvisedFileRegion.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterRunner.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/YarnResourceAllocator.java CHANGES.txt tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterTask.java tajo-project/pom.xml tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMasterClientService.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoAsyncDispatcher.java tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/MiniTajoYarnCluster.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/YarnTaskRunnerLauncherImpl.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainer.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/TajoWorkerResourceManager.java tajo-core/tajo-core-pullserver/src/main/java/org/apache/tajo/pullserver/FadvisedChunkedFile.java tajo-core/tajo-core-backend/src/main/proto/TajoWorkerProtocol.proto tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/YarnRMContainerAllocator.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorkerClientService.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainerId.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/util/ApplicationIdUtils.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMaster.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMasterService.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/Query.java

            People

            • Assignee:
              Jihoon Son
              Reporter:
              Hyunsik Choi
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development