[IMPALA-6215] Race between lib_cache and java udf class loading

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 3.0, Impala 2.12.0
• Component/s: Backend, Frontend
• Epic Color: ghx-label-1
• Assignee: Vuk Ercegovac
• Reporter: Vuk Ercegovac
• Created: 18/Nov/17 00:18
• Updated: 18/Jun/18 15:52
• Resolved: 11/Apr/18 15:05

Description

There's a race between the lib_cache (backend) and UdfExecutor's class loading (frontend).

A sequence of events that exposes this race is as follows:

1. Backend hive-udf-call, in Init obtains the cached, local path to the jar containing the java
method to run
2. A function that is contained in the cached jar is dropped
3. The impala server, on a separate thread learns of the dropped function, so blows away the
containing jar
4. Backend hive-udf-call is opened, which constructs the UdfExecutor given the cached path
to the jar that has just been deleted. A classloader is created with that path-- when the
class is resolved, we get a ClassNotFound exception

An example of this sequence shows up occasionally and is the reason for the following flake:
https://issues.apache.org/jira/browse/IMPALA-6092

Issue Links

• breaks: IMPALA-7150 Crash in Reflection::invoke_method()
• is blocked by: IMPALA-6488 Crash in LibCache::GetCacheEntryInternal()
• is depended upon by: IMPALA-4795 TCatalogObjectFromObjectName is broken for functions
• relates to: IMPALA-6092 Flaky test: query_test/test_udfs.py still happening