Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-1303

CDH cannot pass hadoop version check test

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.10.0
    • Component/s: None
    • Labels:
      None

      Description

      getVersion() of CheckHadoopRuntimeVersionRule.java assumes the pattern of Hadoop version as 'xx.xx.xx'.
      However, if CDH is used, version is expressed as 'xx.xx.xx-cdhxx.xx.xx'
      and getVersion() wrongly retrieves last digit of the version
      as 'xx-cdhxx.xx.xx' like '0-chd5.3.0'.

      I think getVersion() should check digit part of the Hadoop version string only.

      1. TAJO-1303_2.patch
        1 kB
        Keuntae Park
      2. TAJO-1303.patch
        0.8 kB
        Keuntae Park

        Activity

        Hide
        sirpkt Keuntae Park added a comment -

        I changed getVersion to ignore non-digit part of hadoop version.
        I checked it works with CDH 5.3.0 cluster.

        If you compile Tajo for CDH 5.3.0, you should add cloudera repository in pom.xml as following diff indicates:

        diff --git pom.xml pom.xml
        index 8e5dd5e..b17ea68 100644
        --- pom.xml
        +++ pom.xml
        @@ -68,6 +68,10 @@
                 <enabled>false</enabled>
               </snapshots>
             </repository>
        +    <repository>
        +      <id>cloudera</id>
        +      <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
        +    </repository>
           </repositories>
         
           <properties>
        
        Show
        sirpkt Keuntae Park added a comment - I changed getVersion to ignore non-digit part of hadoop version. I checked it works with CDH 5.3.0 cluster. If you compile Tajo for CDH 5.3.0, you should add cloudera repository in pom.xml as following diff indicates: diff --git pom.xml pom.xml index 8e5dd5e..b17ea68 100644 --- pom.xml +++ pom.xml @@ -68,6 +68,10 @@ <enabled> false </enabled> </snapshots> </repository> + <repository> + <id>cloudera</id> + <url>https: //repository.cloudera.com/artifactory/cloudera-repos/</url> + </repository> </repositories> <properties>
        Hide
        ykrips Jihun Kang added a comment - - edited

        Thank you for reporting this issue. I did not notified that version string of hadoop common component is varied. Your patch works well on CDH, but could you add a patch for HDP? Version String of hadoop common project on HDP is also completely different from apache hadoop common component. I have found that the version string of HDP is 2.6.0.2.2.0.0-2041, and we can use the first three part of this version string.

        Show
        ykrips Jihun Kang added a comment - - edited Thank you for reporting this issue. I did not notified that version string of hadoop common component is varied. Your patch works well on CDH, but could you add a patch for HDP? Version String of hadoop common project on HDP is also completely different from apache hadoop common component. I have found that the version string of HDP is 2.6.0.2.2.0.0-2041 , and we can use the first three part of this version string.
        Hide
        sirpkt Keuntae Park added a comment -

        Thank you for the comment, Jihun Kang.
        I updated the patch to support HDP also.
        However, I didn't check it works with actual HDP cluster because I use only CDH and Apache clusters.
        If you don't mind, would you check if it works on HDP?

        Show
        sirpkt Keuntae Park added a comment - Thank you for the comment, Jihun Kang . I updated the patch to support HDP also. However, I didn't check it works with actual HDP cluster because I use only CDH and Apache clusters. If you don't mind, would you check if it works on HDP?
        Hide
        ykrips Jihun Kang added a comment -

        Keuntae Park No problems. I will check it and will share the result.

        Show
        ykrips Jihun Kang added a comment - Keuntae Park No problems. I will check it and will share the result.
        Hide
        ykrips Jihun Kang added a comment -

        +1 This patch works well on HDP 2.1.4.0 with hadoop 2.4.0.

        Show
        ykrips Jihun Kang added a comment - +1 This patch works well on HDP 2.1.4.0 with hadoop 2.4.0.
        Hide
        ykrips Jihun Kang added a comment -

        Committed. Thank you for posting this patch.

        Show
        ykrips Jihun Kang added a comment - Committed. Thank you for posting this patch.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-CODEGEN-build #201 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/201/)
        TAJO-1303: CDH cannot pass hadoop version check test (jihun: rev 0024c75e9a36c6c90ab168f002101725f753f945)

        • tajo-common/src/main/java/org/apache/tajo/rule/base/CheckHadoopRuntimeVersionRule.java
        • CHANGES
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #201 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/201/ ) TAJO-1303 : CDH cannot pass hadoop version check test (jihun: rev 0024c75e9a36c6c90ab168f002101725f753f945) tajo-common/src/main/java/org/apache/tajo/rule/base/CheckHadoopRuntimeVersionRule.java CHANGES
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #562 (See https://builds.apache.org/job/Tajo-master-build/562/)
        TAJO-1303: CDH cannot pass hadoop version check test (jihun: rev 0024c75e9a36c6c90ab168f002101725f753f945)

        • tajo-common/src/main/java/org/apache/tajo/rule/base/CheckHadoopRuntimeVersionRule.java
        • CHANGES
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #562 (See https://builds.apache.org/job/Tajo-master-build/562/ ) TAJO-1303 : CDH cannot pass hadoop version check test (jihun: rev 0024c75e9a36c6c90ab168f002101725f753f945) tajo-common/src/main/java/org/apache/tajo/rule/base/CheckHadoopRuntimeVersionRule.java CHANGES

          People

          • Assignee:
            sirpkt Keuntae Park
            Reporter:
            sirpkt Keuntae Park
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development