[HDFS-12628] libhdfs crashes on thread exit for JNI+libhdfs applications

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 3.0.0-alpha3
• Component/s: native
• Assignee: Unassigned
• Reporter: Joe McDonnell
• Created: 10/Oct/17 22:08
• Updated: 07/May/19 21:04

Description

Impala uses libhdfs to access HDFS while also using JNI to run other Java code. Impala currently relies on HDFS's getJNIEnv to get a JNIEnv to interact with the process JVM (which is created by HDFS code). It uses this JNIEnv even for code that is not related to HDFS.

In recent versions of HDFS, getJNIEnv is no longer visible in libhdfs due to HDFS-7879. In HDFS-8474, the proposed solution was for Impala to write its own equivalent (tracked by IMPALA-2029). After implementing an equivalent of getJNIEnv (heavily based on HDFS code, but with distinct names), we are seeing crashes in hdfsThreadDestructor() in threads that use both HDFS and JNI codepaths. The crash shows up under concurrency and does not reproduce in serial execution.

I have distilled it down to a simple testcase that reproduces the issue. It creates a JVM in the main thread (which Impala does at startup), then spawns multiple threads that do basic HDFS and JNI work. I have removed all but the essential steps.

This blocks running Impala on any hadoop version past 2.7 (when HDFS-7879 was merged). Note that exposing getJNIEnv should unblock Impala development if a fix is not forthcoming.