Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: Catalog
    • Labels:
      None

      Description

      Hi, guys~

      I wish to discuss about HCatalogStore.

      HiveMetaStore types consists of three types: embedded, local, remote.
      This type will set up at hive configuration file which named 'hive-site.xml' and HiveMetaStoreClient can use MetaStore by configuration file. So, if Tajo have only to add hive configuration file to classpath, Tajo can use HiveMetaStore with HiveMetaStoreClient.

      But current HCatalogStore set up configurations about HiveMetaStore as follows:

      • hive.metastore.uris
      • hive.metastore.kerberos.principal
      • hive.metastore.local
      • hive.metastore.sasl.enabled

      And Tajo doesn't need 'tajo.catalog.uri' property because 'hive-site.xml' already includes metastore host and port. So, it looks like unnecessary settings.

      What do you think about this suggestion?

      1. TAJO-433_2.patch
        38 kB
        Jaehwa Jung
      2. TAJO-433.patch
        23 kB
        Jaehwa Jung

        Activity

        Hide
        hyunsik Hyunsik Choi added a comment -

        +1 for this idea! That would be more useful in many cases.

        Show
        hyunsik Hyunsik Choi added a comment - +1 for this idea! That would be more useful in many cases.
        Hide
        jihoonson Jihoon Son added a comment -

        The problem looks reasonable.
        But, I have a question that HiveMetaStoreServer is always accessible when users use Hive meta information.
        How about add both ways to be chosen by users?

        Show
        jihoonson Jihoon Son added a comment - The problem looks reasonable. But, I have a question that HiveMetaStoreServer is always accessible when users use Hive meta information. How about add both ways to be chosen by users?
        Hide
        jhkim Jinho Kim added a comment -

        Guys,
        I think we don't need hive metaserver. please refer to https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin

        Show
        jhkim Jinho Kim added a comment - Guys, I think we don't need hive metaserver. please refer to https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin
        Hide
        blrunner Jaehwa Jung added a comment - - edited

        Thanks Jinho.
        I'll update description about HiveMetaStore, and I think to discuss HCatalog API at another JIRA issue.

        Show
        blrunner Jaehwa Jung added a comment - - edited Thanks Jinho. I'll update description about HiveMetaStore, and I think to discuss HCatalog API at another JIRA issue.
        Hide
        jihoonson Jihoon Son added a comment -

        +1 for the idea.
        This will remove unnecessary configurations and codes of catalog clients.

        Show
        jihoonson Jihoon Son added a comment - +1 for the idea. This will remove unnecessary configurations and codes of catalog clients.
        Hide
        blrunner Jaehwa Jung added a comment -

        Thanks guys.
        I'll start to work.

        Show
        blrunner Jaehwa Jung added a comment - Thanks guys. I'll start to work.
        Hide
        blrunner Jaehwa Jung added a comment -

        I'm glad to upload this patch. From now on, tajo can use HiveMetastore with hive-site.xml. But verifying dependencies for hive was both tricky and time-consuming job. I updated tajo as follows:

        1. HCatalogUtil: It made HiveMetaStoreClient using HiveConf. This update affected HCatalogStore and TestHCatalogStore.
        2. pom.xml: some hive dependencies made trouble which load another hive libraries. So, I checked all hive dependencies.
        3. tajo: Previous tajo added all jar files in hive library directory into class path. But some jar files crashed tajo libraries. For example, tajo uses protobuf version 2.5.0, but hive 0.12. uses protobuf version 2.4.1. So, After I verified each jar file, I added it into class path.
        4. tajo-env.sh: I added HIVE_JDBC_DRIVER_DIR which specify hive jdbc driver jar file path because HCatalogStore needed it.

        You can verify this patch as follows:

        * for hive 0.11.0: mvn clean install -Phcatalog-0.11.0
        * for hive 0.12.0: mvn clean install -Phcatalog-0.12.0
        
        Show
        blrunner Jaehwa Jung added a comment - I'm glad to upload this patch. From now on, tajo can use HiveMetastore with hive-site.xml. But verifying dependencies for hive was both tricky and time-consuming job. I updated tajo as follows: HCatalogUtil: It made HiveMetaStoreClient using HiveConf. This update affected HCatalogStore and TestHCatalogStore. pom.xml: some hive dependencies made trouble which load another hive libraries. So, I checked all hive dependencies. tajo: Previous tajo added all jar files in hive library directory into class path. But some jar files crashed tajo libraries. For example, tajo uses protobuf version 2.5.0, but hive 0.12. uses protobuf version 2.4.1. So, After I verified each jar file, I added it into class path. tajo-env.sh: I added HIVE_JDBC_DRIVER_DIR which specify hive jdbc driver jar file path because HCatalogStore needed it. You can verify this patch as follows: * for hive 0.11.0: mvn clean install -Phcatalog-0.11.0 * for hive 0.12.0: mvn clean install -Phcatalog-0.12.0
        Hide
        blrunner Jaehwa Jung added a comment -

        Hi, guys.
        I think this issue is ready to be closed and committed. Can anybody review it?

        Show
        blrunner Jaehwa Jung added a comment - Hi, guys. I think this issue is ready to be closed and committed. Can anybody review it?
        Hide
        jihoonson Jihoon Son added a comment -

        Sorry for the late review.
        As you know, we are testing the review board for the more efficient review.
        Would you upload the patch at RB, please?

        Show
        jihoonson Jihoon Son added a comment - Sorry for the late review. As you know, we are testing the review board for the more efficient review. Would you upload the patch at RB, please?
        Hide
        blrunner Jaehwa Jung added a comment -

        Thanks Jihoon Son.
        I created a review request in RB https://reviews.apache.org/r/16599/.
        Check it, please.

        Show
        blrunner Jaehwa Jung added a comment - Thanks Jihoon Son . I created a review request in RB https://reviews.apache.org/r/16599/ . Check it, please.
        Hide
        blrunner Jaehwa Jung added a comment -

        I uploaded the second patch. Please check it in RB https://reviews.apache.org/r/16599/ again.

        Show
        blrunner Jaehwa Jung added a comment - I uploaded the second patch. Please check it in RB https://reviews.apache.org/r/16599/ again.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-trunk-postcommit #668 (See https://builds.apache.org/job/Tajo-trunk-postcommit/668/)
        TAJO-433: Improve integration with Hive. (jaehwa) (jhjung: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=19a756d75ef693df4f28a42a9cd3634524f3f577)

        • tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStoreClientPool.java
        • tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/pom.xml
        • tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java
        • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
        • tajo-dist/src/main/bin/tajo
        • NOTICE.txt
        • CHANGES.txt
        • tajo-dist/src/main/conf/tajo-env.sh
        • tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java
        • tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-trunk-postcommit #668 (See https://builds.apache.org/job/Tajo-trunk-postcommit/668/ ) TAJO-433 : Improve integration with Hive. (jaehwa) (jhjung: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=19a756d75ef693df4f28a42a9cd3634524f3f577 ) tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStoreClientPool.java tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/pom.xml tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java tajo-dist/src/main/bin/tajo NOTICE.txt CHANGES.txt tajo-dist/src/main/conf/tajo-env.sh tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java
        Hide
        blrunner Jaehwa Jung added a comment -

        Review was completed. So, I just committed the patch.

        Show
        blrunner Jaehwa Jung added a comment - Review was completed. So, I just committed the patch.

          People

          • Assignee:
            blrunner Jaehwa Jung
            Reporter:
            blrunner Jaehwa Jung
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development