[IMPALA-2072] DCHECK(client->num_tmp_reserved_buffers_ == 0) on test_spilling during repeated-runs

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: Impala 2.2
• Labels: resource-management
• Target Version: Impala 2.3.0
• Assignee: Ippokratis Pandis
• Reporter: Ippokratis Pandis
• Created: 16/Jun/15 22:00
• Updated: 20/Dec/15 00:05
• Resolved: 27/Aug/15 07:32

Description

Some times repeated runs fail at test_spilling. It seems that Impalad crashes. The error from .FATAL:

F0616 14:40:45.102733 19161 buffered-block-mgr.cc:261] Check failed: client->num_tmp_reserved_buffers_ == 0 (5 vs. 0) 

This is in:

bool BufferedBlockMgr::TryAcquireTmpReservation(Client* client, int num_buffers) {

TestSpilling.test_spilling[exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: parquet/none]

self = <test_spillling.TestSpilling object at 0x23c0ad0>
vector = <tests.common.test_vector.TestVector object at 0x2217ad0>

    @pytest.mark.execute_serially
    @CustomClusterTestSuite.with_args(
        impalad_args="--read_size=200000",
        catalogd_args="--load_catalog_in_background=false")
    def test_spilling(self, vector):
      new_vector = deepcopy(vector)
      # remove this. the test cases set this explicitly.
      del new_vector.get_value('exec_option')['num_nodes']
>     self.run_test_case('QueryTest/spilling', new_vector)

>       raise ImpalaBeeswaxException(self.__build_error_message(u), u)
E       ImpalaBeeswaxException: ImpalaBeeswaxException:
E        INNER EXCEPTION: <class 'socket.error'>
E        MESSAGE: [Errno 104] Connection reset by peer

The last query executed:

-- executing against localhost:21000

set max_block_mgr_memory=100m;

-- executing against localhost:21000

select count(l1.l_tax)
from
lineitem l1,
lineitem l2,
lineitem l3
where
l1.l_tax < 0.01 and
l2.l_tax < 0.04 and
l1.l_orderkey = l2.l_orderkey and
l1.l_orderkey = l3.l_orderkey and
l1.l_comment = l3.l_comment and
l1.l_shipdate = l3.l_shipdate;

Issue Links

• is related to: IMPALA-2099 BufferedBlockMgr DCHECK client->num_tmp_reserved_buffers_ == 0 failure w/ RM enabled
• relates to: IMPALA-2245 Check failed: client->num_tmp_reserved_buffers_ == 0 (1 vs. 0)