Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4589

Kudu now requires replication factor must be odd; tests broken

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0
    • Fix Version/s: Impala 2.8.0
    • Component/s: Frontend
    • Labels:

      Description

      In testing the latest kudu build in the Impala toolchain, I discovered that Kudu now requires the replication factor to be odd. We need to fix tests (some set it to 2) and check if any FE/Catalog handling may be necessary.

      13:57:01 -- Test range partitioning with overlapping partitions
      13:57:01 create table simple_range_with_overlapping (id int, name string, valf float, vali bigint,
      13:57:01   primary key (id, name)) distribute by range (id)
      13:57:01   (partition values <= 10, partition values < 20, partition value = 5) stored as kudu;
      13:57:01 
      13:57:01  TestKuduOperations.test_kudu_stats[exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none] 
      13:57:01 [gw0] linux2 -- Python 2.6.6 /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/bin/../infra/python/env/bin/python
      13:57:01 query_test/test_kudu.py:66: in test_kudu_stats
      13:57:01     self.run_test_case('QueryTest/kudu_stats', vector, use_db=unique_database)
      13:57:01 common/impala_test_suite.py:324: in run_test_case
      13:57:01     result = self.__execute_query(target_impalad_client, query, user=user)
      13:57:01 common/impala_test_suite.py:532: in __execute_query
      13:57:01     return impalad_client.execute(query, user=user)
      13:57:01 common/impala_connection.py:160: in execute
      13:57:01     return self.__beeswax_client.execute(sql_stmt, user=user)
      13:57:01 beeswax/impala_beeswax.py:173: in execute
      13:57:01     handle = self.__execute_query(query_string.strip(), user=user)
      13:57:01 beeswax/impala_beeswax.py:337: in __execute_query
      13:57:01     handle = self.execute_query_async(query_string, user=user)
      13:57:01 beeswax/impala_beeswax.py:333: in execute_query_async
      13:57:01     return self.__do_rpc(lambda: self.imp_service.query(query,))
      13:57:01 beeswax/impala_beeswax.py:458: in __do_rpc
      13:57:01     raise ImpalaBeeswaxException(self.__build_error_message(b), b)
      13:57:01 E   ImpalaBeeswaxException: ImpalaBeeswaxException:
      13:57:01 E    INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
      13:57:01 E    MESSAGE: 
      13:57:01 E   ImpalaRuntimeException: Error creating Kudu table 'impala::test_kudu_stats_8950fab4.simple'
      13:57:01 E   CAUSED BY: NonRecoverableException: illegal replication factor 2 (replication factor must be odd)
      

      Testing w/ native-toolchain & build:
      IMPALA_TOOLCHAIN_BUILD_ID=289-f12b0dd2e9
      IMPALA_KUDU_VERSION=60aa54e

        Activity

        Hide
        tlipcon Todd Lipcon added a comment -

        If you prefer, you could also set --allow_unsafe_replication_factor in your test environment

        Show
        tlipcon Todd Lipcon added a comment - If you prefer, you could also set --allow_unsafe_replication_factor in your test environment
        Hide
        mjacobs Matthew Jacobs added a comment -

        Thanks, though it's easy enough to fix. I also changed analysis to reject it accordingly.

        Show
        mjacobs Matthew Jacobs added a comment - Thanks, though it's easy enough to fix. I also changed analysis to reject it accordingly.
        Hide
        tlipcon Todd Lipcon added a comment -

        I don't know if analysis should be trying to enforce the same rules that the Kudu server is enforcing – otherwise if an admin has changed the setting on the server side, then it wouldn't take effect because of the rejection inside Impala. (same goes for all of the other verification that Kudu does server-side on table creation)

        Show
        tlipcon Todd Lipcon added a comment - I don't know if analysis should be trying to enforce the same rules that the Kudu server is enforcing – otherwise if an admin has changed the setting on the server side, then it wouldn't take effect because of the rejection inside Impala. (same goes for all of the other verification that Kudu does server-side on table creation)
        Hide
        mjacobs Matthew Jacobs added a comment -

        commit 5188f879a7940f96dcaf3892f08c0aa76f58abe2
        Author: Matthew Jacobs <mj@cloudera.com>
        Date: Sat Dec 3 09:58:20 2016 -0800

        IMPALA-4477: Bump Kudu version to latest master (60aa54e)

        Bumps the toolchain version to get a newer Kudu build.

        Also fixes test failures resulting from changes in Kudu.
        Notably error strings have changed (IMPALA-4590) and the
        number of replicas must be odd (IMPALA-4589).

        Note: The toolchain binaries starting with this build are
        now using the toolchain binutils rather than the system
        binutils.

        Testing: private exhaustive build.

        Change-Id: If1912f058c240fbe82b06f77e31add7755289be1
        Reviewed-on: http://gerrit.cloudera.org:8080/5369
        Reviewed-by: Matthew Jacobs <mj@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        mjacobs Matthew Jacobs added a comment - commit 5188f879a7940f96dcaf3892f08c0aa76f58abe2 Author: Matthew Jacobs <mj@cloudera.com> Date: Sat Dec 3 09:58:20 2016 -0800 IMPALA-4477 : Bump Kudu version to latest master (60aa54e) Bumps the toolchain version to get a newer Kudu build. Also fixes test failures resulting from changes in Kudu. Notably error strings have changed ( IMPALA-4590 ) and the number of replicas must be odd ( IMPALA-4589 ). Note: The toolchain binaries starting with this build are now using the toolchain binutils rather than the system binutils. Testing: private exhaustive build. Change-Id: If1912f058c240fbe82b06f77e31add7755289be1 Reviewed-on: http://gerrit.cloudera.org:8080/5369 Reviewed-by: Matthew Jacobs <mj@cloudera.com> Tested-by: Internal Jenkins
        Hide
        jrussell John Russell added a comment -

        The docs already mentioned the requirement for replication factor to be odd. I'll add it also as a potential Incompatible Change item in the gerrit review for release notes:

        https://gerrit.cloudera.org/#/c/5668/

        Show
        jrussell John Russell added a comment - The docs already mentioned the requirement for replication factor to be odd. I'll add it also as a potential Incompatible Change item in the gerrit review for release notes: https://gerrit.cloudera.org/#/c/5668/

          People

          • Assignee:
            mjacobs Matthew Jacobs
            Reporter:
            mjacobs Matthew Jacobs
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development