Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4636

Add support for SLES12 for Impala/Kudu integration

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.7.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Backend
    • Labels:

      Description

      During remote cluster testing, we discovered that Impala integration is not supported on SLES12, even though SLES12 is on the list of Kudu's supported OS's.

      (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql):
      INSERT into TABLE functional_kudu.alltypes
      SELECT id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col,
             cast(timestamp_col as string), year, month
      FROM functional.alltypes
      
      Data Loading from Impala failed with error: ImpalaBeeswaxException:
       Query aborted:
      Kudu is not supported on this operating system.
      

      The problem is that we're not currently building a SLES12 client for Kudu. This is not a critical issue, but it should be added at some point.

        Activity

        Hide
        lv Lars Volker added a comment -

        I'm changing the "Fix Version/s" to 2.9 as part of a bulk edit. If you think this is wrong, please set the version to the correct one.

        Show
        lv Lars Volker added a comment - I'm changing the "Fix Version/s" to 2.9 as part of a bulk edit. If you think this is wrong, please set the version to the correct one.
        Hide
        dknupp David Knupp added a comment -

        We have a clean test run on a SLES12 SP1 cluster running the latest CDH 5.10 build.

        Show
        dknupp David Knupp added a comment - We have a clean test run on a SLES12 SP1 cluster running the latest CDH 5.10 build.
        Hide
        dknupp David Knupp added a comment -

        Just confirmed that Kudu data is successfully loading onto a SLES12 cluster running the latest nightly build

        (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql):
        INSERT into TABLE functional_kudu.alltypes
        SELECT id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col,
               cast(timestamp_col as string), year, month
        FROM functional.alltypes
        
        (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql):
        INSERT into TABLE functional_kudu.alltypessmall
        SELECT id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col,
               cast(timestamp_col as string), year, month
        FROM functional.alltypessmall
        
        (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql):
        INSERT INTO TABLE functional_kudu.alltypestiny
        SELECT id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col,
               cast(timestamp_col as string), year, month
        FROM functional.alltypestiny
        
        (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql):
        INSERT into TABLE functional_kudu.alltypesagg_idx
        SELECT row_number() over (order by year, month, id, day),
               id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col,
               double_col, date_string_col, string_col,
               cast(timestamp_col as string), year, month, day
        FROM functional.alltypesagg
        
        (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql):
        INSERT into TABLE functional_kudu.alltypesaggnonulls
        SELECT id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col,
               double_col, date_string_col, string_col,
               cast(timestamp_col as string), year, month, day
        FROM functional.alltypesaggnonulls
        
        (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql):
        INSERT INTO TABLE functional_kudu.dimtbl SELECT * FROM functional.dimtbl
        
        (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql):
        INSERT INTO TABLE functional_kudu.jointbl SELECT * FROM functional.jointbl
        
        (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql):
        INSERT INTO TABLE functional_kudu.tinytable SELECT * FROM functional.tinytable
        
        (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql):
        INSERT INTO TABLE functional_kudu.tinyinttable SELECT * FROM functional.tinyinttable
        
        (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql):
        INSERT INTO TABLE functional_kudu.nulltable select * from functional.nulltable
        
        (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql):
        INSERT INTO TABLE functional_kudu.nullescapedtable select * from functional.nulltable
        
        (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql):
        INSERT INTO TABLE functional_kudu.zipcode_incomes SELECT * FROM functional.zipcode_incomes
        
        (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql):
        
        Invalidating Metadata
        Data loading for workload 'functional-query' completed in: 1461.32s
        
        Show
        dknupp David Knupp added a comment - Just confirmed that Kudu data is successfully loading onto a SLES12 cluster running the latest nightly build (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql): INSERT into TABLE functional_kudu.alltypes SELECT id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, cast(timestamp_col as string), year, month FROM functional.alltypes (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql): INSERT into TABLE functional_kudu.alltypessmall SELECT id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, cast(timestamp_col as string), year, month FROM functional.alltypessmall (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql): INSERT INTO TABLE functional_kudu.alltypestiny SELECT id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, cast(timestamp_col as string), year, month FROM functional.alltypestiny (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql): INSERT into TABLE functional_kudu.alltypesagg_idx SELECT row_number() over (order by year, month, id, day), id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, cast(timestamp_col as string), year, month, day FROM functional.alltypesagg (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql): INSERT into TABLE functional_kudu.alltypesaggnonulls SELECT id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, cast(timestamp_col as string), year, month, day FROM functional.alltypesaggnonulls (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql): INSERT INTO TABLE functional_kudu.dimtbl SELECT * FROM functional.dimtbl (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql): INSERT INTO TABLE functional_kudu.jointbl SELECT * FROM functional.jointbl (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql): INSERT INTO TABLE functional_kudu.tinytable SELECT * FROM functional.tinytable (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql): INSERT INTO TABLE functional_kudu.tinyinttable SELECT * FROM functional.tinyinttable (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql): INSERT INTO TABLE functional_kudu.nulltable select * from functional.nulltable (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql): INSERT INTO TABLE functional_kudu.nullescapedtable select * from functional.nulltable (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql): INSERT INTO TABLE functional_kudu.zipcode_incomes SELECT * FROM functional.zipcode_incomes (load-functional-query-exhaustive-impala-load-generated-kudu-none-none.sql): Invalidating Metadata Data loading for workload 'functional-query' completed in: 1461.32s
        Hide
        dknupp David Knupp added a comment -
        Show
        dknupp David Knupp added a comment - New patch in play: https://gerrit.cloudera.org/#/c/5519/
        Hide
        dknupp David Knupp added a comment -

        I still have my instance, and can do the Impala side. Off the top of your head, is this only file that needs to be fixed?

        Show
        dknupp David Knupp added a comment - I still have my instance, and can do the Impala side. Off the top of your head, is this only file that needs to be fixed?
        Hide
        mjacobs Matthew Jacobs added a comment -

        Yikes, my bad. I somehow thought that this was the right value, but clearly was working too fast. Thanks for catching! We should update that string to suselinux12. Do you have the sles box still and would you be able to run the build on it? I have a few other things in flight at the moment, perhaps you can shepherd through a proper fix in the Impala side? The toolchain bits should be there now in AWS, I did verify that in my last toolchain build.

        Show
        mjacobs Matthew Jacobs added a comment - Yikes, my bad. I somehow thought that this was the right value, but clearly was working too fast. Thanks for catching! We should update that string to suselinux12. Do you have the sles box still and would you be able to run the build on it? I have a few other things in flight at the moment, perhaps you can shepherd through a proper fix in the Impala side? The toolchain bits should be there now in AWS, I did verify that in my last toolchain build.
        Hide
        dknupp David Knupp added a comment -

        Matthew Jacobs Did you test this on an actual SLES 12 instance? I tried to run tests on a SLES12 SP1 cluster, and I got the same error. I found at least one obvious problem.

        It looks like impala-config.sh tries to resolve the OS using:

        DISTRO_VERSION="$(lsb_release -sir 2>&1)"
        # ...snip...
        
        DISTRO_VERSION="$(tr -d ' \n' <<< "$DISTRO_VERSION" | cut -d. -f1 | tr "A-Z" "a-z")"
        case "$DISTRO_VERSION" in
          centos6 | centos7 | debian7 | debian8 | sles12 | ubuntu* )
              KUDU_IS_SUPPORTED=true;;
        esac
        

        However, on sles1sp1 at any rate, evaluating DISTRO_VERSION using the above code results in suselinux12.

        Show
        dknupp David Knupp added a comment - Matthew Jacobs Did you test this on an actual SLES 12 instance? I tried to run tests on a SLES12 SP1 cluster, and I got the same error. I found at least one obvious problem. It looks like impala-config.sh tries to resolve the OS using: DISTRO_VERSION= "$(lsb_release -sir 2>&1)" # ...snip... DISTRO_VERSION= "$(tr -d ' \n' <<< " $DISTRO_VERSION " | cut -d. -f1 | tr " A-Z " " a-z ")" case "$DISTRO_VERSION" in centos6 | centos7 | debian7 | debian8 | sles12 | ubuntu* ) KUDU_IS_SUPPORTED= true ;; esac However, on sles1sp1 at any rate, evaluating DISTRO_VERSION using the above code results in suselinux12 .
        Hide
        mjacobs Matthew Jacobs added a comment -

        patch in a patch seems to work:
        https://gerrit.cloudera.org/#/c/5454/

        Show
        mjacobs Matthew Jacobs added a comment - patch in a patch seems to work: https://gerrit.cloudera.org/#/c/5454/

          People

          • Assignee:
            mjacobs Matthew Jacobs
            Reporter:
            dknupp David Knupp
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development