[IMPALA-11457] Ozone parallelism reduced when backends are co-located

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 4.1.0
• Fix Version/s: Impala 4.2.0, Impala 4.1.1
• Component/s: Backend
• Epic Color: ghx-label-11
• Assignee: Michael Smith
• Reporter: Michael Smith
• Created: 26/Jul/22 17:31
• Updated: 30/Aug/22 12:10
• Resolved: 03/Aug/22 16:05

Description

When Impala and Ozone backends are co-located (at the same IP address), access to Ozone will use a local filesystem access path. It's supposed to assign each backend a separate disk ID so access happens in parallel, but a bug in assigning queues when storage ID is unknown results in all Ozone backends sharing the same queue. This results in reduced parallelism as the queue uses a single I/O thread.

To make Impala see Ozone as co-located in a dev environment, start Impala with --hostname=localhost (which is already the default for Ozone).

Issue Links

• fixes: IMPALA-11448 Always assign Ozone I/O to remote thread group to improve performance
• is related to: IMPALA-10213 Handle block location for Ozone
• relates to: IMPALA-10375 Lock down which filesystem types use the file handle cache
• relates to: IMPALA-11448 Always assign Ozone I/O to remote thread group to improve performance