[IMPALA-7465] Hit memory limit in TestScanMemLimit.test_kudu_scan_mem_usage on ASAN build

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 3.1.0
• Fix Version/s: Impala 3.1.0
• Component/s: Backend
• Labels: broken-build
• Target Version: Impala 3.1.0
• Epic Color: ghx-label-6
• Assignee: Tim Armstrong
• Reporter: Joe McDonnell
• Created: 20/Aug/18 19:35
• Updated: 21/Sep/18 23:49
• Resolved: 21/Aug/18 21:49

Description

An ASAN build encountered a failure in test_mem_usage_scaling.TestScanMemLimit.test_kudu_scan_mem_usage() due to hitting the memory limit:

E   ImpalaBeeswaxException: ImpalaBeeswaxException:
E    Query aborted:Memory limit exceeded: Failed to allocate tuple buffer
E   KUDU_SCAN_NODE (id=0) could not allocate 110.00 KB without exceeding limit.
E   Error occurred on backend impala-ec2-centos74-m5-4xlarge-ondemand-0104.vpc.cloudera.com:22001 by fragment 8745b4d457dcf8ab:79a096ab00000003
E   Memory left in process limit: 11.53 GB
E   Memory left in query limit: 7.95 KB
E   Query(8745b4d457dcf8ab:79a096ab00000000): Limit=4.00 MB Reservation=0 ReservationLimit=0 OtherMemory=3.99 MB Total=3.99 MB Peak=3.99 MB
E     Fragment 8745b4d457dcf8ab:79a096ab00000003: Reservation=0 OtherMemory=3.99 MB Total=3.99 MB Peak=3.99 MB
E       SORT_NODE (id=1): Total=342.00 KB Peak=342.00 KB
E       KUDU_SCAN_NODE (id=0): Total=3.64 MB Peak=3.64 MB
E         Queued Batches: Total=3.63 MB Peak=3.63 MB
E       KrpcDataStreamSender (dst_id=2): Total=1.16 KB Peak=1.16 KB
E       CodeGen: Total=10.89 KB Peak=1.15 MB
E   
E   Memory limit exceeded: Failed to allocate tuple buffer
E   KUDU_SCAN_NODE (id=0) could not allocate 110.00 KB without exceeding limit.
E   Error occurred on backend impala-ec2-centos74-m5-4xlarge-ondemand-0104.vpc.cloudera.com:22001 by fragment 8745b4d457dcf8ab:79a096ab00000003
E   Memory left in process limit: 11.53 GB
E   Memory left in query limit: 7.95 KB
E   Query(8745b4d457dcf8ab:79a096ab00000000): Limit=4.00 MB Reservation=0 ReservationLimit=0 OtherMemory=3.99 MB Total=3.99 MB Peak=3.99 MB
E     Fragment 8745b4d457dcf8ab:79a096ab00000003: Reservation=0 OtherMemory=3.99 MB Total=3.99 MB Peak=3.99 MB
E       SORT_NODE (id=1): Total=342.00 KB Peak=342.00 KB
E       KUDU_SCAN_NODE (id=0): Total=3.64 MB Peak=3.64 MB
E         Queued Batches: Total=3.63 MB Peak=3.63 MB
E       KrpcDataStreamSender (dst_id=2): Total=1.16 KB Peak=1.16 KB
E       CodeGen: Total=10.89 KB Peak=1.15 MB

This test was introduced recently as part of IMPALA-7096.

Issue Links

• is related to: IMPALA-7096 Ensure no memory limit exceeded regressions from IMPALA-4835 because of non-reserved memory
• is related to: IMPALA-7600 Mem limit exceeded in test_kudu_scan_mem_usage