[IMPALA-9177] TestTpchQuery.test_tpch query 18 on Kudu sometimes hits memory limit on dockerised tests

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 3.4.0
• Fix Version/s: Impala 3.4.0
• Component/s: Backend
• Labels: broken-build flaky
• Epic Color: ghx-label-13
• Assignee: Joe McDonnell
• Reporter: Joe McDonnell
• Created: 20/Nov/19 23:01
• Updated: 22/Nov/19 00:07
• Resolved: 22/Nov/19 00:07

Description

A few upstream GVO jobs have seen TestTpchQuery.test_tpch Kudu Q18 hit a memory limit:

E   ImpalaBeeswaxException: ImpalaBeeswaxException:
E    Query aborted:Memory limit exceeded: Cannot perform aggregation at aggregator with id 4. Failed to allocate 25 bytes for intermediate tuple.
E   Fragment 734a83b2dc908a93:15cf4ff60000000c could not allocate 25.00 B without exceeding limit.
E   Error occurred on backend e3cb8b541195:22000 by fragment 734a83b2dc908a93:15cf4ff60000000c
E   Memory left in process limit: 6.46 GB
E   Memory left in query limit: 12.37 KB
E   Query(734a83b2dc908a93:15cf4ff600000000): Limit=256.00 MB Reservation=189.38 MB ReservationLimit=204.80 MB OtherMemory=66.61 MB Total=255.99 MB Peak=256.00 MB
E     Unclaimed reservations: Reservation=147.38 MB OtherMemory=0 Total=147.38 MB Peak=181.38 MB
E     Fragment 734a83b2dc908a93:15cf4ff600000011: Reservation=0 OtherMemory=93.02 KB Total=93.02 KB Peak=1.70 MB
E       SORT_NODE (id=9): Total=4.00 KB Peak=4.00 KB
E       AGGREGATION_NODE (id=16): Reservation=0 OtherMemory=61.12 KB Total=61.12 KB Peak=61.12 KB
E         GroupingAggregator 0: Total=61.12 KB Peak=61.12 KB
E           Exprs: Total=61.12 KB Peak=61.12 KB
E       EXCHANGE_NODE (id=15): Reservation=0 OtherMemory=0 Total=0 Peak=0
E         KrpcDeferredRpcs: Total=0 Peak=0
E       KrpcDataStreamSender (dst_id=17): Total=1.97 KB Peak=1.97 KB
E       CodeGen: Total=17.93 KB Peak=1.63 MB
E     Fragment 734a83b2dc908a93:15cf4ff60000000e: Reservation=0 OtherMemory=55.28 MB Total=55.28 MB Peak=55.28 MB
E       AGGREGATION_NODE (id=8): Reservation=0 OtherMemory=61.12 KB Total=61.12 KB Peak=61.12 KB
E         GroupingAggregator 0: Total=61.12 KB Peak=61.12 KB
E           Exprs: Total=61.12 KB Peak=61.12 KB
E       HASH_JOIN_NODE (id=7): Total=58.25 KB Peak=58.25 KB
E         Exprs: Total=13.12 KB Peak=13.12 KB
E         Hash Join Builder (join_node_id=7): Total=13.12 KB Peak=13.12 KB
E           Hash Join Builder (join_node_id=7) Exprs: Total=13.12 KB Peak=13.12 KB
E       HASH_JOIN_NODE (id=6): Total=50.25 KB Peak=50.25 KB
E         Exprs: Total=13.12 KB Peak=13.12 KB
E         Hash Join Builder (join_node_id=6): Total=13.12 KB Peak=13.12 KB
E           Hash Join Builder (join_node_id=6) Exprs: Total=13.12 KB Peak=13.12 KB
E       HASH_JOIN_NODE (id=5): Total=42.25 KB Peak=42.25 KB
E         Exprs: Total=13.12 KB Peak=13.12 KB
E         Hash Join Builder (join_node_id=5): Total=13.12 KB Peak=13.12 KB
E           Hash Join Builder (join_node_id=5) Exprs: Total=13.12 KB Peak=13.12 KB
E       EXCHANGE_NODE (id=10): Reservation=10.20 MB OtherMemory=29.22 KB Total=10.22 MB Peak=10.22 MB
E         KrpcDeferredRpcs: Total=29.22 KB Peak=29.22 KB
E       EXCHANGE_NODE (id=11): Reservation=17.38 MB OtherMemory=35.71 KB Total=17.42 MB Peak=17.42 MB
E         KrpcDeferredRpcs: Total=35.71 KB Peak=35.71 KB
E       EXCHANGE_NODE (id=12): Reservation=10.35 MB OtherMemory=0 Total=10.35 MB Peak=10.35 MB
E         KrpcDeferredRpcs: Total=0 Peak=0
E       AGGREGATION_NODE (id=14): Reservation=0 OtherMemory=17.12 KB Total=17.12 KB Peak=17.12 KB
E         GroupingAggregator 0: Total=17.12 KB Peak=17.12 KB
E           Exprs: Total=17.12 KB Peak=17.12 KB
E       EXCHANGE_NODE (id=13): Reservation=11.09 MB OtherMemory=0 Total=11.09 MB Peak=11.09 MB
E         KrpcDeferredRpcs: Total=0 Peak=0
E       KrpcDataStreamSender (dst_id=15): Total=5.91 KB Peak=5.91 KB
E       CodeGen: Total=5.97 MB Peak=5.97 MB
E     Fragment 734a83b2dc908a93:15cf4ff600000002: Reservation=0 OtherMemory=1.60 MB Total=1.60 MB Peak=1.63 MB
E       KUDU_SCAN_NODE (id=2): Total=1.49 MB Peak=1.51 MB
E         Queued Batches: Total=1.49 MB Peak=1.51 MB
E       KrpcDataStreamSender (dst_id=10): Total=82.57 KB Peak=106.57 KB
E       CodeGen: Total=1.00 KB Peak=166.00 KB
E     Fragment 734a83b2dc908a93:15cf4ff600000006: Reservation=0 OtherMemory=8.00 MB Total=8.00 MB Peak=8.03 MB
E       KUDU_SCAN_NODE (id=1): Total=7.80 MB Peak=7.93 MB
E         Queued Batches: Total=7.80 MB Peak=7.93 MB
E       KrpcDataStreamSender (dst_id=11): Total=69.35 KB Peak=93.35 KB
E       CodeGen: Total=1.09 KB Peak=175.00 KB
E     Fragment 734a83b2dc908a93:15cf4ff60000000c: Reservation=42.00 MB OtherMemory=1.64 MB Total=43.64 MB Peak=43.70 MB
E       AGGREGATION_NODE (id=4): Reservation=42.00 MB OtherMemory=42.12 KB Total=42.04 MB Peak=42.04 MB
E         GroupingAggregator 0: Reservation=42.00 MB OtherMemory=17.12 KB Total=42.02 MB Peak=42.02 MB
E           Exprs: Total=17.12 KB Peak=17.12 KB
E       KUDU_SCAN_NODE (id=3): Total=1.49 MB Peak=1.51 MB
E         Queued Batches: Total=1.49 MB Peak=1.51 MB
E       KrpcDataStreamSender (dst_id=13): Total=75.35 KB Peak=99.35 KB
E       CodeGen: Total=5.29 KB Peak=596.50 KB
E     Fragment 734a83b2dc908a93:15cf4ff600000008: Reservation=0 OtherMemory=0 Total=0 Peak=1.56 MB
E       KUDU_SCAN_NODE (id=0): Total=0 Peak=1.46 MB
E       KrpcDataStreamSender (dst_id=12): Total=0 Peak=15.35 KB
E       CodeGen: Total=0 Peak=0

There have been a few occurrences, so far on dockerised-tests:

ubuntu-16.04-dockerised-tests/1577

ubuntu-16.04-dockerised-tests/1655

ubuntu-16.04-dockerised-tests/1649