[SPARK-20848] Dangling threads when reading parquet files in local mode - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 2.1.1, 2.2.0
Fix Version/s: 2.1.2, 2.2.0
Component/s: Input/Output, SQL
Labels:
None

Description

On each call to spark.read.parquet, a new ForkJoinPool is created. One of the threads in the pool is kept in the WAITING state, and never stopped, which leads to unbounded growth in number of threads.

This behavior is a regression from v2.1.0.

Reproducible example:

val spark = SparkSession
  .builder()
  .appName("test")
  .master("local")
  .getOrCreate()
while(true) {
  spark.read.parquet("/path/to/file")
  Thread.sleep(5000)
}