Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5695

test_role_update test leaves the mini-cluster in a bad state after execution

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: Impala 2.10.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Epic Color:
      ghx-label-7

      Description

      test_role_update right now is possibly the last test that gets executed when running all the tests.

      Adding any test after test_role_update that restarts impala mini-cluster will hang.
      Ex: I ran test_role_privilege_case after test_role_update and it failed with this error -

      anuj@anuj-OptiPlex-9020:~/Impala$ ps auxw | grep impalad
      anuj     25516 49.8  0.0      0     0 pts/6    Z+   12:21   0:04 [impalad] <defunct>    <<<<<< zombie process 
      anuj     25792  0.6  0.0  56356 12160 pts/6    S+   12:21   0:00 /home/anuj/Impala/bin/../infra/python/env/bin/python /home/anuj/Impala/bin/start-impala-cluster.py --cluster_size=3 --num_coordinators=3 --log_dir=/tmp/ --log_level=1 --impalad_args="--server_name=server1"  --state_store_args="--statestore_update_frequency_ms=300"  --catalogd_args="--sentry_config=/home/anuj/Impala/fe/src/test/resources/sentry-site.xml --sentry_catalog_polling_frequency_s=1" 
      

      Here is the stack trace -

      Traceback (most recent call last):
        File "/home/anuj/Impala/bin/start-impala-cluster.py", line 365, in <module>
          kill_cluster_processes(force=options.force_kill)
        File "/home/anuj/Impala/bin/start-impala-cluster.py", line 143, in kill_cluster_processes
          kill_matching_processes(binaries, force)
        File "/home/anuj/Impala/bin/start-impala-cluster.py", line 163, in kill_matching_processes
          process.pid, KILL_TIMEOUT_IN_SECONDS))
      RuntimeError: Unable to kill impalad (pid 25516) after 240 seconds.
      MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
      ERROR
      ===================================================== short test summary info ======================================================
      ERROR authorization/test_grant_revoke.py::TestGrantRevoke::()::test_role_privilege_case[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      
      ============================================================== ERRORS ==============================================================
       ERROR at setup of TestGrantRevoke.test_role_privilege_case[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none] 
      authorization/test_grant_revoke.py:46: in setup_method
          super(TestGrantRevoke, self).setup_method(method)
      common/custom_cluster_test_suite.py:103: in setup_method
          self._start_impala_cluster(cluster_args)
      common/custom_cluster_test_suite.py:129: in _start_impala_cluster
          check_call(cmd + options, close_fds=True)
      /usr/lib/python2.7/subprocess.py:540: in check_call
          raise CalledProcessError(retcode, cmd)
      E   CalledProcessError: Command '['/home/anuj/Impala/bin/start-impala-cluster.py', '--cluster_size=3', '--num_coordinators=3', '--log_dir=/tmp/', '--log_level=1', '--impalad_args="--server_name=server1" ', '--state_store_args="--statestore_update_frequency_ms=300" ', '--catalogd_args="--sentry_config=/home/anuj/Impala/fe/src/test/resources/sentry-site.xml --sentry_catalog_polling_frequency_s=1" ']' returned non-zero exit status 1
      !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
      ====================================== 3 passed, 1 pytest-warnings, 1 error in 364.06 seconds ======================================
      anuj@anuj-OptiPlex-9020:~/Impala/tests$ 
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                anujphadke Anuj Phadke
                Reporter:
                anujphadke Anuj Phadke
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: