[IMPALA-12635] Tests fail with "Memory is likely oversubscribed"

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 4.4.0
• Fix Version/s: Impala 4.4.0
• Component/s: Backend
• Epic Color: ghx-label-6
• Assignee: Michael Smith
• Reporter: Andrew Sherman
• Created: 14/Dec/23 22:27
• Updated: 15/Dec/23 18:45
• Resolved: 15/Dec/23 18:44

Description

3 different tests failed trying to allocate memory

E   OperationalError: Failed to get minimum memory reservation of 94.11 MB on daemon impala-ec2-centos79-m6i-4xlarge-xldisk-0682.vpc.cloudera.com:27000 for query a94be0ce8a16884d:cef55b1a00000000 due to following error: Memory limit exceeded: Could not allocate memory while trying to increase reservation.
E   Query(a94be0ce8a16884d:cef55b1a00000000) could not allocate 94.11 MB without exceeding limit.
E   Error occurred on backend impala-ec2-centos79-m6i-4xlarge-xldisk-0682.vpc.cloudera.com:27000
E   Memory left in process limit: 1.24 GB
E   Query(a94be0ce8a16884d:cef55b1a00000000): Reservation=0 ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0
E   Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error.

E   ImpalaBeeswaxException: ImpalaBeeswaxException:
E    Query aborted:Failed to get minimum memory reservation of 4.03 MB on daemon impala-ec2-centos79-m6i-4xlarge-xldisk-0682.vpc.cloudera.com:27000 for query 3e4b9c2bd24a8c30:5db8f3e100000000 due to following error: Memory limit exceeded: Could not allocate memory while trying to increase reservation.
E   Query(3e4b9c2bd24a8c30:5db8f3e100000000) could not allocate 4.03 MB without exceeding limit.
E   Error occurred on backend impala-ec2-centos79-m6i-4xlarge-xldisk-0682.vpc.cloudera.com:27000
E   Memory left in process limit: 1.20 GB
E   Query(3e4b9c2bd24a8c30:5db8f3e100000000): Reservation=0 ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0
E   Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error.

E   OperationalError: Failed to get minimum memory reservation of 284.18 MB on daemon impala-ec2-centos79-m6i-4xlarge-xldisk-0682.vpc.cloudera.com:27000 for query 054cab730f0605ca:6e70e3dd00000000 due to following error: Memory limit exceeded: Could not allocate memory while trying to increase reservation.
E   Query(054cab730f0605ca:6e70e3dd00000000) could not allocate 284.18 MB without exceeding limit.
E   Error occurred on backend impala-ec2-centos79-m6i-4xlarge-xldisk-0682.vpc.cloudera.com:27000
E   Memory left in process limit: 1.34 GB
E   Query(054cab730f0605ca:6e70e3dd00000000): Reservation=0 ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0
E   Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error.

wzhou I'm randomly picking on you to own this. This has only happened once.

Seen in tests like

• query_test.test_queries.TestQueries.test_intersect
• query_test.test_queries.TestQueries.test_except
• query_test.test_scanners.TestScannersAllTableFormatsWithLimit.test_limit
• query_test.test_join_queries.TestJoinQueries.test_outer_joins
• query_test.test_iceberg.TestIcebergV2Table.test_update_basic
• query_test.test_scanners.TestScannersVirtualColumns.test_virtual_column_input_file_name

Issue Links

• is related to: IMPALA-12641 Impala underestimates LLVM memory use