[IMPALA-5770] test_kudu_insert failed with "Remote error: Service unavailable: Soft memory limit exceeded"

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Duplicate
• Affects Version/s: Impala 2.10.0
• Fix Version/s: Product Backlog
• Component/s: Backend
• Labels: broken-build kudu
• Target Version: Impala 2.10.0
• Epic Color: ghx-label-9
• Assignee: Matthew Jacobs
• Reporter: Michael Ho
• Created: 07/Aug/17 01:03
• Updated: 07/Aug/17 22:59
• Resolved: 07/Aug/17 22:59

Description

query_test/test_kudu.py:81: in test_kudu_insert
    self.run_test_case('QueryTest/kudu_insert', vector, use_db=unique_database)
common/impala_test_suite.py:390: in run_test_case
    result = self.__execute_query(target_impalad_client, query, user=user)
common/impala_test_suite.py:598: in __execute_query
    return impalad_client.execute(query, user=user)
common/impala_connection.py:160: in execute
    return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:173: in execute
    handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:339: in __execute_query
    self.wait_for_completion(handle)
beeswax/impala_beeswax.py:359: in wait_for_completion
    raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
E   ImpalaBeeswaxException: ImpalaBeeswaxException:
E    Query aborted:Kudu error(s) reported, first error: Timed out: Failed to write batch of 33707 ops to tablet 3f5d8cdd0bf842359df0f3ed8467b3dc after 572 attempt(s): Failed to write to server: (no server available): Write(tablet: 3f5d8cdd0bf842359df0f3ed8467b3dc, num_ops: 33707, num_attempts: 572) passed its deadline: Remote error: Service unavailable: Soft memory limit exceeded (at 99.95% of capacity)
E   
E   Key already present in Kudu table 'impala::test_kudu_insert_886525c4.kudu_test'. (1 of 15 similar)
E   Error in Kudu table 'impala::test_kudu_insert_886525c4.kudu_test': Timed out: Failed to write batch of 33707 ops to tablet 3f5d8cdd0bf842359df0f3ed8467b3dc after 572 attempt(s): Failed to write to server: (no server available): Write(tablet: 3f5d8cdd0bf842359df0f3ed8467b3dc, num_ops: 33707, num_attempts: 572) passed its deadline: Remote error: Service unavailable: Soft memory limit exceeded (at 99.95% of capacity) (1 of 33707 similar)

mjacobs, can you please take a first look and re-assign to Kudu if necessary ?

Issue Links

• duplicates: KUDU-2083 MaintenanceManager running_op_ count not decremented if MaintenanceOp::Prepare() fails