[HBASE-19848] Zookeeper thread leaks in hbase-spark bulkLoad method

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 3.0.0-alpha-1
• Component/s: spark, Zookeeper
• Labels: performance

Environment

hbase-spark-1.2.0-cdh5.12.1 version

spark 1.6

• Hadoop Flags: Reviewed
• Flags: Patch, Important
• Assignee: Key Hutu
• Reporter: Key Hutu
• Created: 23/Jan/18 13:10
• Updated: 17/Aug/20 17:45
• Resolved: 02/Feb/18 13:45

Description

In hbase-spark project, HBaseContext provides bulkload methond for loading spark rdd data to hbase easily.But when i using it frequently， the program will throw "cannot create native thread" exception.

using pstack command in spark driver process , the thread num is increasing 

using jstack, named "main-SendThread" and "main-EventThread"  thread so many

It seems like that , connection created before bulkload ,but close method uninvoked at last