Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-10559

TestScratchLimit seems flaky

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 4.0.0
    • Fix Version/s: Impala 4.0.0
    • Component/s: Backend
    • Labels:
    • Epic Color:
      ghx-label-6

      Description

       

      The TestScratchLimit testcase runs fine alone. But if runs TestScratchDir first, then runs TestScratchLimit, it will always fail as follow shows unless restarts the services (e.g. using "./buildall.sh -noclean -notests -format -start_minicluster -start_impala_cluster").

      query_test/test_scratch_limit.py::TestScratchLimit::test_with_low_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      query_test/test_scratch_limit.py::TestScratchLimit::test_without_specifying_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      query_test/test_scratch_limit.py::TestScratchLimit::test_with_high_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      query_test/test_scratch_limit.py::TestScratchLimit::test_with_zero_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      query_test/test_scratch_limit.py::TestScratchLimit::test_with_unlimited_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      query_test/test_scratch_limit.py::TestScratchLimit::test_with_zero_scratch_limit_no_memory_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      [gw1] PASSED query_test/test_scratch_limit.py::TestScratchLimit::test_with_zero_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      [gw5] FAILED query_test/test_scratch_limit.py::TestScratchLimit::test_with_low_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      [gw4] FAILED query_test/test_scratch_limit.py::TestScratchLimit::test_with_high_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      [gw2] FAILED query_test/test_scratch_limit.py::TestScratchLimit::test_without_specifying_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      [gw3] FAILED query_test/test_scratch_limit.py::TestScratchLimit::test_with_unlimited_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      [gw0] PASSED query_test/test_scratch_limit.py::TestScratchLimit::test_with_zero_scratch_limit_no_memory_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      ------------------------------------------- generated xml file: /home/yida/Impala/logs/ee_tests/results/TEST-impala-parallel.xml --------------------------------------------
      ========================================================================== short test summary info ==========================================================================
      FAIL query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_low_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      FAIL query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_high_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      FAIL query_test/test_scratch_limit.py::TestScratchLimit::()::test_without_specifying_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      FAIL query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_unlimited_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      ================================================================================= FAILURES ==================================================================================
       TestScratchLimit.test_with_low_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      [gw5] linux2 -- Python 2.7.16 /home/yida/Impala/bin/../infra/python/env-gcc7.5.0/bin/python
      query_test/test_scratch_limit.py:90: in test_with_low_scratch_limit
       assert expected_error % scratch_limit_in_bytes in str(e)
      E assert ('Scratch space limit of %s bytes exceeded' % 25165824) in 'ImpalaBeeswaxException:\n Query aborted:Could not create files in any configured scratch directories (--scratch_dirs=...ous errors that may have prevented creating or writing scratch files. The following directories were at capacity: \n\n'
      E + where 'ImpalaBeeswaxException:\n Query aborted:Could not create files in any configured scratch directories (--scratch_dirs=...ous errors that may have prevented creating or writing scratch files. The following directories were at capacity: \n\n' = str(ImpalaBeeswaxException())
      --------------------------------------------------------------------------- Captured stderr setup ---------------------------------------------------------------------------
      SET client_identifier=query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_low_scratch_limit[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_threshol;
      -- connecting to: localhost:21000
      -- connecting to localhost:21050 with impyla
      -- 2021-03-03 07:12:08,115 INFO MainThread: Closing active operation
      -- connecting to localhost:28000 with impyla
      -- 2021-03-03 07:12:08,140 INFO MainThread: Closing active operation
      --------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------
      SET client_identifier=query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_low_scratch_limit[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_threshol;
      SET scratch_limit=24m;
      SET batch_size=0;
      SET num_nodes=0;
      SET disable_codegen_rows_threshold=5000;
      SET disable_codegen=False;
      SET abort_on_error=1;
      SET exec_single_node_rows_threshold=0;
      SET buffer_pool_limit=32m;
      -- executing against localhost:21000
      
       select o_orderdate, o_custkey, o_comment
       from tpch.orders
       order by o_orderdate
       ;
      -- 2021-03-03 07:12:08,329 INFO MainThread: Started query f44b4a50bec6b10d:e45a053d00000000
       TestScratchLimit.test_with_high_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      [gw4] linux2 -- Python 2.7.16 /home/yida/Impala/bin/../infra/python/env-gcc7.5.0/bin/python
      query_test/test_scratch_limit.py:74: in test_with_high_scratch_limit
       self.execute_query_expect_success(self.client, self.spilling_sort_query, exec_option)
      common/impala_test_suite.py:809: in wrapper
       return function(*args, **kwargs)
      common/impala_test_suite.py:817: in execute_query_expect_success
       result = cls.__execute_query(impalad_client, query, query_options, user)
      common/impala_test_suite.py:918: in __execute_query
       return impalad_client.execute(query, user=user)
      common/impala_connection.py:205: in execute
       return self.__beeswax_client.execute(sql_stmt, user=user)
      beeswax/impala_beeswax.py:187: in execute
       handle = self.__execute_query(query_string.strip(), user=user)
      beeswax/impala_beeswax.py:365: in __execute_query
       self.wait_for_finished(handle)
      beeswax/impala_beeswax.py:386: in wait_for_finished
       raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
      E ImpalaBeeswaxException: ImpalaBeeswaxException:
      E Query aborted:Could not create files in any configured scratch directories (--scratch_dirs=) on backend 'yida-OptiPlex-7060:27001'. 0 of scratch is currently in use by this Impala Daemon (0 by this query). See logs for previous errors that may have prevented creating or writing scratch files. The following directories were at capacity:
      --------------------------------------------------------------------------- Captured stderr setup ---------------------------------------------------------------------------
      SET client_identifier=query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_high_scratch_limit[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_thresho;
      -- connecting to: localhost:21000
      -- connecting to localhost:21050 with impyla
      -- 2021-03-03 07:12:08,114 INFO MainThread: Closing active operation
      -- connecting to localhost:28000 with impyla
      -- 2021-03-03 07:12:08,136 INFO MainThread: Closing active operation
      --------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------
      SET client_identifier=query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_high_scratch_limit[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_thresho;
      SET scratch_limit=500m;
      SET batch_size=0;
      SET num_nodes=0;
      SET disable_codegen_rows_threshold=5000;
      SET disable_codegen=False;
      SET abort_on_error=1;
      SET exec_single_node_rows_threshold=0;
      SET buffer_pool_limit=32m;
      -- executing against localhost:21000
      
       select o_orderdate, o_custkey, o_comment
       from tpch.orders
       order by o_orderdate
       ;
      -- 2021-03-03 07:12:08,329 INFO MainThread: Started query 814fccc9c1242331:83738b0800000000
       TestScratchLimit.test_without_specifying_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      [gw2] linux2 -- Python 2.7.16 /home/yida/Impala/bin/../infra/python/env-gcc7.5.0/bin/python
      query_test/test_scratch_limit.py:118: in test_without_specifying_scratch_limit
       self.execute_query_expect_success(self.client, self.spilling_sort_query, exec_option)
      common/impala_test_suite.py:809: in wrapper
       return function(*args, **kwargs)
      common/impala_test_suite.py:817: in execute_query_expect_success
       result = cls.__execute_query(impalad_client, query, query_options, user)
      common/impala_test_suite.py:918: in __execute_query
       return impalad_client.execute(query, user=user)
      common/impala_connection.py:205: in execute
       return self.__beeswax_client.execute(sql_stmt, user=user)
      beeswax/impala_beeswax.py:187: in execute
       handle = self.__execute_query(query_string.strip(), user=user)
      beeswax/impala_beeswax.py:365: in __execute_query
       self.wait_for_finished(handle)
      beeswax/impala_beeswax.py:386: in wait_for_finished
       raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
      E ImpalaBeeswaxException: ImpalaBeeswaxException:
      E Query aborted:Could not create files in any configured scratch directories (--scratch_dirs=) on backend 'yida-OptiPlex-7060:27002'. 0 of scratch is currently in use by this Impala Daemon (0 by this query). See logs for previous errors that may have prevented creating or writing scratch files. The following directories were at capacity:
      --------------------------------------------------------------------------- Captured stderr setup ---------------------------------------------------------------------------
      SET client_identifier=query_test/test_scratch_limit.py::TestScratchLimit::()::test_without_specifying_scratch_limit[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disable_codegen':False;'abort_on_error':1;'exec_single_node_row;
      -- connecting to: localhost:21000
      -- connecting to localhost:21050 with impyla
      -- 2021-03-03 07:12:08,114 INFO MainThread: Closing active operation
      -- connecting to localhost:28000 with impyla
      -- 2021-03-03 07:12:08,137 INFO MainThread: Closing active operation
      --------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------
      SET client_identifier=query_test/test_scratch_limit.py::TestScratchLimit::()::test_without_specifying_scratch_limit[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disable_codegen':False;'abort_on_error':1;'exec_single_node_row;
      SET batch_size=0;
      SET num_nodes=0;
      SET disable_codegen_rows_threshold=5000;
      SET disable_codegen=False;
      SET abort_on_error=1;
      SET exec_single_node_rows_threshold=0;
      SET buffer_pool_limit=32m;
      -- executing against localhost:21000
      
       select o_orderdate, o_custkey, o_comment
       from tpch.orders
       order by o_orderdate
       ;
      -- 2021-03-03 07:12:08,329 INFO MainThread: Started query 944af93b81ee62cf:ab33f08800000000
       TestScratchLimit.test_with_unlimited_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
      [gw3] linux2 -- Python 2.7.16 /home/yida/Impala/bin/../infra/python/env-gcc7.5.0/bin/python
      query_test/test_scratch_limit.py:110: in test_with_unlimited_scratch_limit
       self.execute_query_expect_success(self.client, self.spilling_sort_query, exec_option)
      common/impala_test_suite.py:809: in wrapper
       return function(*args, **kwargs)
      common/impala_test_suite.py:817: in execute_query_expect_success
       result = cls.__execute_query(impalad_client, query, query_options, user)
      common/impala_test_suite.py:918: in __execute_query
       return impalad_client.execute(query, user=user)
      common/impala_connection.py:205: in execute
       return self.__beeswax_client.execute(sql_stmt, user=user)
      beeswax/impala_beeswax.py:187: in execute
       handle = self.__execute_query(query_string.strip(), user=user)
      beeswax/impala_beeswax.py:365: in __execute_query
       self.wait_for_finished(handle)
      beeswax/impala_beeswax.py:386: in wait_for_finished
       raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
      E ImpalaBeeswaxException: ImpalaBeeswaxException:
      E Query aborted:Could not create files in any configured scratch directories (--scratch_dirs=) on backend 'yida-OptiPlex-7060:27001'. 0 of scratch is currently in use by this Impala Daemon (0 by this query). See logs for previous errors that may have prevented creating or writing scratch files. The following directories were at capacity:
      --------------------------------------------------------------------------- Captured stderr setup ---------------------------------------------------------------------------
      SET client_identifier=query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_unlimited_scratch_limit[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_th;
      -- connecting to: localhost:21000
      -- connecting to localhost:21050 with impyla
      -- 2021-03-03 07:12:08,114 INFO MainThread: Closing active operation
      -- connecting to localhost:28000 with impyla
      -- 2021-03-03 07:12:08,146 INFO MainThread: Closing active operation
      --------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------
      SET client_identifier=query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_unlimited_scratch_limit[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_th;
      SET scratch_limit=-1;
      SET batch_size=0;
      SET num_nodes=0;
      SET disable_codegen_rows_threshold=5000;
      SET disable_codegen=False;
      SET abort_on_error=1;
      SET exec_single_node_rows_threshold=0;
      SET buffer_pool_limit=32m;
      -- executing against localhost:21000
      
       select o_orderdate, o_custkey, o_comment
       from tpch.orders
       order by o_orderdate
       ;
      -- 2021-03-03 07:12:08,329 INFO MainThread: Started query 5c492cfd62287b03:f9c3370b00000000
      ==================================================================== 4 failed, 2 passed in 14.11 seconds ====================================================================
      Running TestExecutor with args: ['--ignore', 'query_test', '--ignore', 'catalog_service', '--ignore', 'hs2', '--ignore', 'experiments', '--ignore', 'data_errors', '--ignore', 'performance', '--ignore', 'beeswax', '--ignore', 'authorization', '--ignore', 'metadata', '--ignore', 'shell', '--ignore', 'benchmark', '--ignore', 'infra', '--ignore', 'statestore', '--ignore', 'failure', '--ignore', 'webserver', '--ignore', 'comparison', '--ignore', 'stress', '--ignore', 'util', '--ignore', 'custom_cluster', '--ignore', 'common', '--ignore', 'observability', '--ignore', 'unittests', '--resultlog', '/home/yida/Impala/logs/ee_tests/results/TEST-impala-verify-metrics.log', '--junitxml', '/home/yida/Impala/logs/ee_tests/results/TEST-impala-verify-metrics.xml', 'verifiers/test_verify_metrics.py']
      rootLoggerLevel = INFO
      

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                rizaon Riza Suminto
                Reporter:
                baggio000 Yida Wu
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: