Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.9.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Catalog
    • Labels:
      None

      Issue Links

        Activity

        Hide
        tarmstrong Tim Armstrong added a comment -

        IMPALA-5184: build fe against both Hive 1 & 2 APIs

        This adds a compatibility shim layer with Hive 1 and Hive 2
        implementations. The version-specific code lives in
        fe/src/compat-hive-$IMPALA_HIVE_MAJOR_VERSION and
        common/thrift/hive-$IMPALA_HIVE_MAJOR_VERSION-api/

        The shim adds wrapper methods to handle differing method signatures
        and and config variables that changed slightly.

        Some thrift classes were also moved from the 'cli' to 'rpc' package.
        We work around these by implementing subclasses with the same name
        in a different package for compatibility or by implementing shim
        methods that operate on the classes. We also need to change the
        package in the TCLIService.thrift, which is done with a
        search-and-replace.

        Also avoid the sticky config variable problem with some of the source
        paths by requiring an _OVERRIDE suffix on the variable to override it
        from the environment.

        Testing:
        Made sure that I could build Impala on master as normal, and also
        with the following config overrides in bin/impala-config-local.sh:

        export IMPALA_HADOOP_VERSION=3.0.0-alpha1-cdh6.x-SNAPSHOT
        export IMPALA_HBASE_VERSION=2.0.0-cdh6.x-SNAPSHOT
        export IMPALA_HIVE_VERSION=2.1.0-cdh6.x-SNAPSHOT
        export IMPALA_SENTRY_VERSION=1.5.1-cdh6.x-SNAPSHOT
        export IMPALA_PARQUET_VERSION=1.5.0-cdh6.x-SNAPSHOT

        I manually assembled the dependencies by copying the following files
        from the Hive 2 source and from a Hadoop 3 build:
        $CDH_COMPONENTS_HOME/hive-2.1.0-cdh6.x-SNAPSHOT/src/metastore/if/hive_metastore.thrift
        $CDH_COMPONENTS_HOME/hadoop-3.0.0-alpha1-cdh6.x-SNAPSHOT/lib/native/*
        $CDH_COMPONENTS_HOME/hadoop-3.0.0-alpha1-cdh6.x-SNAPSHOT/include/hdfs.h

        Change-Id: Ifbc265281c04fe3136bc3c920dbac966742ce09a
        Reviewed-on: http://gerrit.cloudera.org:8080/5538
        Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
        Tested-by: Impala Public Jenkins

        Show
        tarmstrong Tim Armstrong added a comment - IMPALA-5184 : build fe against both Hive 1 & 2 APIs This adds a compatibility shim layer with Hive 1 and Hive 2 implementations. The version-specific code lives in fe/src/compat-hive-$IMPALA_HIVE_MAJOR_VERSION and common/thrift/hive-$IMPALA_HIVE_MAJOR_VERSION-api/ The shim adds wrapper methods to handle differing method signatures and and config variables that changed slightly. Some thrift classes were also moved from the 'cli' to 'rpc' package. We work around these by implementing subclasses with the same name in a different package for compatibility or by implementing shim methods that operate on the classes. We also need to change the package in the TCLIService.thrift, which is done with a search-and-replace. Also avoid the sticky config variable problem with some of the source paths by requiring an _OVERRIDE suffix on the variable to override it from the environment. Testing: Made sure that I could build Impala on master as normal, and also with the following config overrides in bin/impala-config-local.sh: export IMPALA_HADOOP_VERSION=3.0.0-alpha1-cdh6.x-SNAPSHOT export IMPALA_HBASE_VERSION=2.0.0-cdh6.x-SNAPSHOT export IMPALA_HIVE_VERSION=2.1.0-cdh6.x-SNAPSHOT export IMPALA_SENTRY_VERSION=1.5.1-cdh6.x-SNAPSHOT export IMPALA_PARQUET_VERSION=1.5.0-cdh6.x-SNAPSHOT I manually assembled the dependencies by copying the following files from the Hive 2 source and from a Hadoop 3 build: $CDH_COMPONENTS_HOME/hive-2.1.0-cdh6.x-SNAPSHOT/src/metastore/if/hive_metastore.thrift $CDH_COMPONENTS_HOME/hadoop-3.0.0-alpha1-cdh6.x-SNAPSHOT/lib/native/* $CDH_COMPONENTS_HOME/hadoop-3.0.0-alpha1-cdh6.x-SNAPSHOT/include/hdfs.h Change-Id: Ifbc265281c04fe3136bc3c920dbac966742ce09a Reviewed-on: http://gerrit.cloudera.org:8080/5538 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Impala Public Jenkins
        Hide
        tarmstrong Tim Armstrong added a comment -

        It looks like the patch missed at least one instance of StatsSetupConst.STATS_GENERATED_VIA_STATS_TASK - probably a conflict with a catalog change that wasn't caught when rebasing.

        Show
        tarmstrong Tim Armstrong added a comment - It looks like the patch missed at least one instance of StatsSetupConst.STATS_GENERATED_VIA_STATS_TASK - probably a conflict with a catalog change that wasn't caught when rebasing.
        Hide
        joemcdonnell Joe McDonnell added a comment -

        commit 330558e46ee2a6e2e4d423f21fdb908fa6450d72
        Author: Joe McDonnell <joemcdonnell@cloudera.com>
        Date: Tue Jun 20 09:45:40 2017 -0700

        IMPALA-5536: Fix TCLIService thrift compilation on Hive 2

        This fixes some issues with Impala compilation on Hive 2:

        1. It fixes TCLIService.thrift generation by changing
        common/thrift/CMakeLists.txt to create the directory
        before generating the file and adding a dependency
        to generate TCLIService.thrift before compiling the
        thrift files.
        2. It modifies CatalogOpExecutor::applyAlterTable to
        use the MetastoreShim.

        With these changes, Impala builds successfully with
        Hive 2 in my environment and on automated builds.

        Change-Id: I7e1477993ee3ccddd236609efec7bb23f20a7b66
        Reviewed-on: http://gerrit.cloudera.org:8080/7231
        Tested-by: Impala Public Jenkins
        Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>

        Show
        joemcdonnell Joe McDonnell added a comment - commit 330558e46ee2a6e2e4d423f21fdb908fa6450d72 Author: Joe McDonnell <joemcdonnell@cloudera.com> Date: Tue Jun 20 09:45:40 2017 -0700 IMPALA-5536 : Fix TCLIService thrift compilation on Hive 2 This fixes some issues with Impala compilation on Hive 2: 1. It fixes TCLIService.thrift generation by changing common/thrift/CMakeLists.txt to create the directory before generating the file and adding a dependency to generate TCLIService.thrift before compiling the thrift files. 2. It modifies CatalogOpExecutor::applyAlterTable to use the MetastoreShim. With these changes, Impala builds successfully with Hive 2 in my environment and on automated builds. Change-Id: I7e1477993ee3ccddd236609efec7bb23f20a7b66 Reviewed-on: http://gerrit.cloudera.org:8080/7231 Tested-by: Impala Public Jenkins Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>

          People

          • Assignee:
            joemcdonnell Joe McDonnell
            Reporter:
            tarmstrong Tim Armstrong
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development