Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: build
    • Labels:

      Description

      Hadoop 2.2.0 has been released. We need to bump up hadoop to 2.2.0. This hadoop version uses protobuf-2.5.0. So, Tajo also needs to bump up protobuf-2.5.0.

      1. TAJO-135.patch
        75 kB
        Jihoon Son
      2. TAJO-135_3.patch
        74 kB
        Jihoon Son
      3. TAJO-135_2.patch
        75 kB
        Jihoon Son

        Issue Links

          Activity

          Hyunsik Choi created issue -
          Hyunsik Choi made changes -
          Field Original Value New Value
          Status Open [ 1 ] In Progress [ 3 ]
          Hyunsik Choi made changes -
          Fix Version/s 0.3-incubating [ 12324253 ]
          Fix Version/s 0.2-incubating [ 12324252 ]
          Jihoon Son made changes -
          Summary Bump up hadoop to 2.1.0-beta Bump up hadoop to 2.2.0
          Jihoon Son made changes -
          Description Hadoop 2.1.0-beta has been released. We need to bump up hadoop to 2.1.0-beta. This hadoop version uses protobuf-2.5.0. So, Tajo also needs to bump up protobuf-2.5.0. Hadoop 2.2.0 has been released. We need to bump up hadoop to 2.2.0. This hadoop version uses protobuf-2.5.0. So, Tajo also needs to bump up protobuf-2.5.0.
          Jihoon Son made changes -
          Assignee Hyunsik Choi [ hyunsik ] Jihoon Son [ jihoonson ]
          Hide
          Jihoon Son added a comment -

          I'll start this issue.

          Show
          Jihoon Son added a comment - I'll start this issue.
          Hyunsik Choi made changes -
          Link This issue is related to TAJO-328 [ TAJO-328 ]
          Hide
          Jihoon Son added a comment -

          Since Hive uses protobuf-2.4.1, I cannot proceed this issue until the Hive dependency is separated from the tajo-core dependency.

          Show
          Jihoon Son added a comment - Since Hive uses protobuf-2.4.1, I cannot proceed this issue until the Hive dependency is separated from the tajo-core dependency.
          Hyunsik Choi made changes -
          Link This issue is blocked by TAJO-336 [ TAJO-336 ]
          Hyunsik Choi made changes -
          Priority Major [ 3 ] Critical [ 2 ]
          Hide
          Jihoon Son added a comment -

          This patch passed 'mvn clean verify' successfully.

          Show
          Jihoon Son added a comment - This patch passed 'mvn clean verify' successfully.
          Jihoon Son made changes -
          Attachment TAJO-135.patch [ 12618182 ]
          Hide
          Jihoon Son added a comment -

          I uploaded the second patch after removing some commented out codes.

          Show
          Jihoon Son added a comment - I uploaded the second patch after removing some commented out codes.
          Jihoon Son made changes -
          Attachment TAJO-135_2.patch [ 12618183 ]
          Jihoon Son made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Hide
          Jaehwa Jung added a comment -

          I failed at common module unit tests as follows:

          Results :
          
          Tests in error: 
            testCreate(org.apache.tajo.datum.protobuf.TestProtobufDatum): This is supposed to be overridden by subclasses.
            testWriteLoadProtoFromFile(org.apache.tajo.util.TestFileUtils): This is supposed to be overridden by subclasses.
            testWriteLoadProtoFromStream(org.apache.tajo.util.TestFileUtils): This is supposed to be overridden by subclasses.
            testWriteLoadProtoFromPath(org.apache.tajo.util.TestFileUtils): This is supposed to be overridden by subclasses.
          
          Tests run: 132, Failures: 0, Errors: 4, Skipped: 0
          
          Show
          Jaehwa Jung added a comment - I failed at common module unit tests as follows: Results : Tests in error: testCreate(org.apache.tajo.datum.protobuf.TestProtobufDatum): This is supposed to be overridden by subclasses. testWriteLoadProtoFromFile(org.apache.tajo.util.TestFileUtils): This is supposed to be overridden by subclasses. testWriteLoadProtoFromStream(org.apache.tajo.util.TestFileUtils): This is supposed to be overridden by subclasses. testWriteLoadProtoFromPath(org.apache.tajo.util.TestFileUtils): This is supposed to be overridden by subclasses. Tests run: 132, Failures: 0, Errors: 4, Skipped: 0
          Hide
          Jihoon Son added a comment -

          Thanks, JaeHwa.
          The above message means that the protobuf version is mismatched.
          Did you test with protobuf 2.5.0?

          Show
          Jihoon Son added a comment - Thanks, JaeHwa. The above message means that the protobuf version is mismatched. Did you test with protobuf 2.5.0?
          Hide
          Hyunsik Choi added a comment -

          JaeHwa,

          The error may be caused by old protoc version. Since hadoop-2.2.0, we should use protoc 2.5.0. I've verified 'mvn clean install' with protoc 2.5.0.

          Show
          Hyunsik Choi added a comment - JaeHwa, The error may be caused by old protoc version. Since hadoop-2.2.0, we should use protoc 2.5.0. I've verified 'mvn clean install' with protoc 2.5.0.
          Hide
          Jaehwa Jung added a comment -

          Sorry, Jihoon.

          I tested with protobuf 2.4.1.
          I'll try test again with protobuf 2.5.0.

          Show
          Jaehwa Jung added a comment - Sorry, Jihoon. I tested with protobuf 2.4.1. I'll try test again with protobuf 2.5.0.
          Hide
          Henry Saputra added a comment -

          HI Jihoon, since you modify a lot of the files would you mind updating the default comment from IntelliJ Idea "//To change body of implemented methods use File | Settings | File Templates." to something more useful. Thanks.

          Show
          Henry Saputra added a comment - HI Jihoon, since you modify a lot of the files would you mind updating the default comment from IntelliJ Idea "//To change body of implemented methods use File | Settings | File Templates." to something more useful. Thanks.
          Hide
          Jihoon Son added a comment -

          Thanks, Henry.
          Those comments are not added in this patch, but have been existed before this issue.
          In my opinion, since this issue just focuses on bumping up the hadoop version, it would be better that those comments are changed to something more useful in other issues.

          Show
          Jihoon Son added a comment - Thanks, Henry. Those comments are not added in this patch, but have been existed before this issue. In my opinion, since this issue just focuses on bumping up the hadoop version, it would be better that those comments are changed to something more useful in other issues.
          Hide
          Jihoon Son added a comment -

          Please review the patch.

          Show
          Jihoon Son added a comment - Please review the patch.
          Hide
          Hyunsik Choi added a comment -

          I'm sorry for the review. I'm reviewing the patch.

          Show
          Hyunsik Choi added a comment - I'm sorry for the review. I'm reviewing the patch.
          Hide
          Hyunsik Choi added a comment -

          The patch looks good for me. Some classes include unused imports caused by changes. It would be better if you remove them.

          In addition, I only tested it on a local cluster. After I test it on a real cluster, I'll give +1 for this patch. Thanks!

          Show
          Hyunsik Choi added a comment - The patch looks good for me. Some classes include unused imports caused by changes. It would be better if you remove them. In addition, I only tested it on a local cluster. After I test it on a real cluster, I'll give +1 for this patch. Thanks!
          Hide
          Jihoon Son added a comment -

          Thanks, Hyunsik.
          I uploaded the third patch after clearing unused imports.

          Show
          Jihoon Son added a comment - Thanks, Hyunsik. I uploaded the third patch after clearing unused imports.
          Jihoon Son made changes -
          Attachment TAJO-135_3.patch [ 12620146 ]
          Hide
          Jinho Kim added a comment -

          ++1
          Great Jihoon!!
          I've tested the patch on real cluster. it worked well

          • De/Compression
          • TPC-H 1, 3
          Show
          Jinho Kim added a comment - ++1 Great Jihoon!! I've tested the patch on real cluster. it worked well De/Compression TPC-H 1, 3
          Hide
          Hyunsik Choi added a comment -

          +1
          I also verified the patch.

          Show
          Hyunsik Choi added a comment - +1 I also verified the patch.
          Hide
          Jihoon Son added a comment -

          Thanks guys.
          I've just committed the patch.

          Show
          Jihoon Son added a comment - Thanks guys. I've just committed the patch.
          Jihoon Son made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Tajo-trunk-postcommit #637 (See https://builds.apache.org/job/Tajo-trunk-postcommit/637/)
          TAJO-135: Bump up hadoop to 2.2.0. (jihoon) (jihoonson: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=ecc749e811b9f978c9d9bb2cd9bfee6f795653df)

          • tajo-core/tajo-core-backend/src/main/proto/InternalTypes.proto
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/YarnContainerProxy.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/YarnTajoResourceManager.java
          • tajo-core/tajo-core-backend/src/main/proto/TajoMasterProtocol.proto
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorker.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMaster.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/AbstractTaskScheduler.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoResourceAllocator.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
          • tajo-core/tajo-core-backend/pom.xml
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterManagerService.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/GlobalEngine.java
          • tajo-common/src/main/proto/yarn_protos.proto
          • tajo-core/tajo-core-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java
          • tajo-core/tajo-core-pullserver/src/main/java/org/apache/tajo/pullserver/PullServerAuxService.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TaskRunnerManager.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TaskRunner.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryInProgress.java
          • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TestTajoIds.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/AbstractResourceAllocator.java
          • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TajoTestingCluster.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorkerManagerService.java
          • BUILDING.txt
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryJobManager.java
          • tajo-common/src/main/proto/Security.proto
          • tajo-core/tajo-core-pullserver/src/main/java/org/apache/tajo/pullserver/FadvisedFileRegion.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterRunner.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/YarnResourceAllocator.java
          • CHANGES.txt
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterTask.java
          • tajo-project/pom.xml
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMasterClientService.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoAsyncDispatcher.java
          • tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto
          • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/MiniTajoYarnCluster.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/YarnTaskRunnerLauncherImpl.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainer.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/TajoWorkerResourceManager.java
          • tajo-core/tajo-core-pullserver/src/main/java/org/apache/tajo/pullserver/FadvisedChunkedFile.java
          • tajo-core/tajo-core-backend/src/main/proto/TajoWorkerProtocol.proto
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/YarnRMContainerAllocator.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorkerClientService.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainerId.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/util/ApplicationIdUtils.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMaster.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMasterService.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/Query.java
          Show
          Hudson added a comment - FAILURE: Integrated in Tajo-trunk-postcommit #637 (See https://builds.apache.org/job/Tajo-trunk-postcommit/637/ ) TAJO-135 : Bump up hadoop to 2.2.0. (jihoon) (jihoonson: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=ecc749e811b9f978c9d9bb2cd9bfee6f795653df ) tajo-core/tajo-core-backend/src/main/proto/InternalTypes.proto tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/YarnContainerProxy.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/YarnTajoResourceManager.java tajo-core/tajo-core-backend/src/main/proto/TajoMasterProtocol.proto tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorker.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMaster.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/AbstractTaskScheduler.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoResourceAllocator.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java tajo-core/tajo-core-backend/pom.xml tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterManagerService.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/GlobalEngine.java tajo-common/src/main/proto/yarn_protos.proto tajo-core/tajo-core-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java tajo-core/tajo-core-pullserver/src/main/java/org/apache/tajo/pullserver/PullServerAuxService.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TaskRunnerManager.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TaskRunner.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryInProgress.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TestTajoIds.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/AbstractResourceAllocator.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TajoTestingCluster.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorkerManagerService.java BUILDING.txt tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryJobManager.java tajo-common/src/main/proto/Security.proto tajo-core/tajo-core-pullserver/src/main/java/org/apache/tajo/pullserver/FadvisedFileRegion.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterRunner.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/YarnResourceAllocator.java CHANGES.txt tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterTask.java tajo-project/pom.xml tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMasterClientService.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoAsyncDispatcher.java tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/MiniTajoYarnCluster.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/YarnTaskRunnerLauncherImpl.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainer.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/TajoWorkerResourceManager.java tajo-core/tajo-core-pullserver/src/main/java/org/apache/tajo/pullserver/FadvisedChunkedFile.java tajo-core/tajo-core-backend/src/main/proto/TajoWorkerProtocol.proto tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/YarnRMContainerAllocator.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorkerClientService.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainerId.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/util/ApplicationIdUtils.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMaster.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMasterService.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/Query.java

            People

            • Assignee:
              Jihoon Son
              Reporter:
              Hyunsik Choi
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development