[IMPALA-6983] stress test binary search exits if process mem_limit is too low

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 3.1.0
• Component/s: Infrastructure
• Epic Color: ghx-label-8
• Assignee: Michael Brown
• Reporter: Daniel Hecht
• Created: 07/May/18 16:28
• Updated: 18/May/18 05:11
• Resolved: 16/May/18 15:26

Description

This was running stress test on tpch SF=20 and minicluster process mem_limit=7857355161.

2018-05-04 18:25:03,800 18531 MainThread INFO:concurrent_select[1303]:Collecting runtime info for query q5:
select
n_name,
sum(l_extendedprice * (1 - l_discount)) as revenue
from
customer,
orders,
lineitem,
supplier,
nation,
region
where
c_custkey = o_custkey
and l_orderkey = o_orderkey
and l_suppkey = s_suppkey
and c_nationkey = s_nationkey
and s_nationkey = n_nationkey
and n_regionkey = r_regionkey
and r_name = 'ASIA'
and o_orderdate >= '1994-01-01'
and o_orderdate < '1995-01-01'
group by
n_name
order by
revenue desc
2018-05-04 18:25:07,790 18531 MainThread INFO:concurrent_select[1406]:Finding a starting point for binary search
2018-05-04 18:25:07,790 18531 MainThread INFO:concurrent_select[1409]:Next mem_limit: 7493
2018-05-04 18:28:06,380 18531 MainThread WARNING:concurrent_select[1416]:Query couldn't be run even when using all available memory
select
n_name,
sum(l_extendedprice * (1 - l_discount)) as revenue
from
customer,
orders,
lineitem,
supplier,
nation,
region
where
c_custkey = o_custkey
and l_orderkey = o_orderkey
and l_suppkey = s_suppkey
and c_nationkey = s_nationkey
and s_nationkey = n_nationkey
and n_regionkey = r_regionkey
and r_name = 'ASIA'
and o_orderdate >= '1994-01-01'
and o_orderdate < '1995-01-01'
group by
n_name
order by
revenue desc

Traceback (most recent call last):
File "./tests/stress/concurrent_select.py", line 2265, in <module>
main()
File "./tests/stress/concurrent_select.py", line 2162, in main
queries, impala, converted_args, queries_with_runtime_info_by_db_sql_and_options)
File "./tests/stress/concurrent_select.py", line 1879, in populate_all_queries
os.path.join(converted_args.results_dir, PROFILES_DIR))
File "./tests/stress/concurrent_select.py", line 964, in write_runtime_info_profiles
fh.write(profile)
TypeError: expected a string or other character buffer object

I don't understand the details of concurrent_select.py control flow, but it looks like in this case update_runtime_info() won't get called leading to this issue.