[IMPALA-9612] Runtime filter wait longer than it should be

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.0.0, Impala 3.4.1
• Component/s: Backend
• Epic Color: ghx-label-4
• Assignee: Riza Suminto
• Reporter: Riza Suminto
• Created: 07/Apr/20 01:37
• Updated: 05/Feb/21 13:25
• Resolved: 09/Apr/20 17:31

Description

In one of my query profile, I found an info string like this:
Runtime filters: All filters arrived. Waited 59s783ms. Maximum arrival delay: 15s296ms.
If all runtime filters arrived within 15s296ms, then it should not wait until 59s783ms to proceed.

Looking at runtime-filter.cc, It looks like there is a potential race condition in function RuntimeFilter::WaitForArrival().
bool RuntimeFilter::WaitForArrival(int32_t timeout_ms) const {
unique_lock<mutex> l(arrival_mutex_);
while (arrival_time_.Load() == 0)

{ int64_t ms_since_registration = MonotonicMillis() - registration_time_; int64_t ms_remaining = timeout_ms - ms_since_registration; if (ms_remaining <= 0) break; arrival_cv_.WaitFor(l, ms_remaining * MICROS_PER_MILLI); }

return arrival_time_.Load() != 0;
}
Between checking arrival_time_.Load() and calling arrival_cv_.WaitFor(), arrival_cv_ might be already signaled by either RuntimeFilter::SetFilter() or RuntimeFilter::Cancel() because they do not acquire arrival_mutex_ first.

Issue Links

• is related to: IMPALA-5904 Enable ThreadSanitizer for Impala
• relates to: IMPALA-9611 Hang in HandoffToProbesAndWait() for multithreaded join build