[RANGER-3787] Non-daemon threads started by ElasticSearchAuditDestination cause Spark application hanging

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0, 2.4.0
• Component/s: audit
• Assignee: Unassigned
• Reporter: Zhou Yifan
• Created: 14/Jun/22 02:25
• Updated: 08/Nov/22 21:49
• Resolved: 08/Nov/22 21:49

Description

When using kyuubi-spark-authz plugin (which extends RangerBasePlugin) to save audit log to ElasticSearch in Spark application, I found that if Spark application was submitted in local or client mode, it hanged forever even after main thread exited. But if I turn off `xasecure.audit.destination.elasticsearch`, Spark application exits normally.

 

Here is my `ranger-spark-audit.xml`:

<configuration>    
   <property>
        <name>xasecure.audit.destination.elasticsearch</name>
        <value>enabled</value>
    </property>    <property>
        <name>xasecure.audit.destination.elasticsearch.urls</name>
        <value>es-master-1,es-master-2,es-master-3</value>
    </property>    <property>
        <name>xasecure.audit.destination.elasticsearch.port</name>
        <value>9200</value>
    </property>
</configuration> 

In `jstack` output, I found 2 kinds of non-daemon threads started by ElasticSearch RestHighLevelClient:

"I/O dispatcher 1" #64 prio=5 os_prio=31 tid=0x00007f8717d89800 nid=0x14303 runnable [0x00007000054bc000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
        at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
        at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000007808a7b58> (a sun.nio.ch.Util$3)
        - locked <0x00000007808a7b48> (a java.util.Collections$UnmodifiableSet)
        - locked <0x000000078088b318> (a sun.nio.ch.KQueueSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
        at java.lang.Thread.run(Thread.java:750)"pool-22-thread-1" #63 prio=5 os_prio=31 tid=0x00007f86c77d1800 nid=0xb503 runnable [0x00007000053b9000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
        at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
        at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x0000000780832dc0> (a sun.nio.ch.Util$3)
        - locked <0x0000000780832db0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x0000000780832b60> (a sun.nio.ch.KQueueSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:343)
        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
        at java.lang.Thread.run(Thread.java:750) 

 

Reproduce steps

Build kyuubi-spark-authz plugin:

git clone [e-mail]:zhouyifan279/incubator-kyuubi.git
cd incubator-kyuubi 
./build/mvn clean package -pl :kyuubi-spark-authz_2.12 -Pspark-3.1-ranger-3.0 -DskipTests 

Setup plugin in Spark:

1. Dowload and untar spark distribution: https://www.apache.org/dyn/closer.lua/spark/spark-3.1.3/spark-3.1.3-bin-hadoop3.2.tgz
2. Put `incubator-kyuubi/extensions/spark/kyuubi-spark-authz/target/kyuubi-spark-authz_2.12-1.6.0-SNAPSHOT-1-with-ranger-3.0.jar` under `spark-3.1.3-bin-hadoop3.2/jars`
3. Put `ranger-spark-security.xml` and `ranger-spark-audit.xml` under `spark-3.1.3-bin-hadoop3.2/conf`

Run Spark application: 

./spark-3.1.3-bin-hadoop3.2/bin/spark-submit --master local --class org.apache.spark.examples.sql.hive.SparkHiveExample --conf spark.sql.extensions=spark.sql.extensions org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension examples/jars/spark-examples_2.12-3.1.3.jar