Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4453

test_kudu_crud fails with wrong number of results

    Details

      Description

      This failed an ASAN build: http://sandbox.jenkins.cloudera.com/job/impala-asf-master-core-asan/135.

      =================================== FAILURES ===================================
       TestKuduOperations.test_kudu_crud[exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none] 
      [gw3] linux2 -- Python 2.6.6 /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/bin/../infra/python/env/bin/python
      query_test/test_kudu.py:47: in test_kudu_crud
          self.run_test_case('QueryTest/kudu_crud', vector, use_db=unique_database)
      common/impala_test_suite.py:341: in run_test_case
          self.__verify_results_and_errors(vector, test_section, result, use_db)
      common/impala_test_suite.py:235: in __verify_results_and_errors
          replace_filenames_with_placeholder)
      common/test_result_verifier.py:398: in verify_raw_results
          VERIFIER_MAP[verifier](expected, actual)
      common/test_result_verifier.py:231: in verify_query_result_is_equal
          assert expected_results == actual_results
      E   assert Comparing QueryTestResults (expected vs actual):
      E     : 7200 != : 7201
      ---------------------------- Captured stderr setup -----------------------------
      

        Activity

        Hide
        mjacobs Matthew Jacobs added a comment -

        should be resolved by the kudu fix which is now in the Impala build:

        commit 107fc4e9f9fd7d92a227ac4fc1e5820ee44dade1
        Author: Matthew Jacobs <mj@cloudera.com>
        Date: Wed Nov 16 07:39:00 2016 -0800

        IMPALA-4477: Upgrade Kudu version to latest master

        Change the toolchain build and Kudu version to use
        the latest master, using Kudu commit e836ac.

        Change-Id: I49f8582cc3c0f776167fe3decf4236345ba78bd3
        Reviewed-on: http://gerrit.cloudera.org:8080/5106
        Reviewed-by: Matthew Jacobs <mj@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        mjacobs Matthew Jacobs added a comment - should be resolved by the kudu fix which is now in the Impala build: commit 107fc4e9f9fd7d92a227ac4fc1e5820ee44dade1 Author: Matthew Jacobs <mj@cloudera.com> Date: Wed Nov 16 07:39:00 2016 -0800 IMPALA-4477 : Upgrade Kudu version to latest master Change the toolchain build and Kudu version to use the latest master, using Kudu commit e836ac. Change-Id: I49f8582cc3c0f776167fe3decf4236345ba78bd3 Reviewed-on: http://gerrit.cloudera.org:8080/5106 Reviewed-by: Matthew Jacobs <mj@cloudera.com> Tested-by: Internal Jenkins
        Hide
        mjacobs Matthew Jacobs added a comment -

        I think the issue would have been there for a number of weeks, it seems to be an uncommon race in the Kudu client. I've been working with Kudu on it but I don't have a fix yet. Has this come up anywhere else? I suspect it's pretty uncommon because I think this would have been possible to hit since Oct 19th when I added the additional validation. If you're OK with this, I'd prefer to wait a few days to see if we can get a real fix / workaround from Kudu. I'd rather not lose the coverage of these tests entirely since it's pretty important coverage right now while the Kudu paths are churning. If we don't think we can get a satisfying fix/workaround by Friday, or we continue to see this fail, then I'll submit a patch to skip validating these tests.

        Show
        mjacobs Matthew Jacobs added a comment - I think the issue would have been there for a number of weeks, it seems to be an uncommon race in the Kudu client. I've been working with Kudu on it but I don't have a fix yet. Has this come up anywhere else? I suspect it's pretty uncommon because I think this would have been possible to hit since Oct 19th when I added the additional validation. If you're OK with this, I'd prefer to wait a few days to see if we can get a real fix / workaround from Kudu. I'd rather not lose the coverage of these tests entirely since it's pretty important coverage right now while the Kudu paths are churning. If we don't think we can get a satisfying fix/workaround by Friday, or we continue to see this fail, then I'll submit a patch to skip validating these tests.
        Hide
        henryr Henry Robinson added a comment -

        Is IMPALA-3710 the culprit? Shall we xfail the test, or revert the patch? (xfailing sounds better, because this isn't a product bug in Impala that we know of).

        Show
        henryr Henry Robinson added a comment - Is IMPALA-3710 the culprit? Shall we xfail the test, or revert the patch? (xfailing sounds better, because this isn't a product bug in Impala that we know of).
        Hide
        mjacobs Matthew Jacobs added a comment -

        This appears to be flaky, it passed in the last run.

        After investigating, I believe it to be a bug in the Kudu error reporting which makes us believe there is 1 less error than there should be. I filed https://issues.apache.org/jira/browse/KUDU-1743 and will think about what we can do to unblock tests in the meantime.

        Show
        mjacobs Matthew Jacobs added a comment - This appears to be flaky, it passed in the last run. After investigating, I believe it to be a bug in the Kudu error reporting which makes us believe there is 1 less error than there should be. I filed https://issues.apache.org/jira/browse/KUDU-1743 and will think about what we can do to unblock tests in the meantime.

          People

          • Assignee:
            mjacobs Matthew Jacobs
            Reporter:
            henryr Henry Robinson
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development