[SPARK-23022] Spark Thrift Server always cache resource issues

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.2.1
• Component/s: Deploy

Environment

CentOS6.x
Spark2.x
JDK1.8

• Assignee: Unassigned
• Reporter: hereTac
• Created: 10/Jan/18 02:14
• Updated: 10/Jan/18 03:35
• Resolved: 10/Jan/18 03:35

Description

Hi. I use the Thrift Server for SparkSQL . I queried muiltle queries.I deploy the Spark on Yarn.
When I finish my query.Thrift Server always cache the Yarn Resources. Any suggests will be helpful.

------------------------------------------------------------------------------------
Here is the img .
------------------------------------------------------------------------------------




------------------------------------------------------------------------------------
Here is the Spark Conf .
------------------------------------------------------------------------------------

spark.dynamicAllocation.minExecutors 1
spark.dynamicAllocation.maxExecutors 500
spark.dynamicAllocation.enabled true
spark.shuffle.service.enabled true
spark.executor.instances 2
spark.executor.memory   6g
#serializer
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.kryoserializer.buffer.max 512m
#spark.kryo.unsafe true
spark.kryo.referenceTracking false
spark.rdd.compress true
spark.memory.offHeap.enabled true
spark.memory.offHeap.size 1g

spark.yarn.archive hdfs://ns/data1/hadooptmp/spark2.2.1/jars
spark.yarn.am.memory 2g
spark.driver.memory 4g
spark.driver.maxResultSize 2g

#SPARK SQL 
spark.sql.shuffle.partitions 500
spark.sql.statistics.fallBackToHdfs true
spark.sql.orc.filterPushdown true
spark.sql.autoBroadcastJoinThreshold 104857600
spark.sql.adaptive.enabled true

spark.history.fs.logDirectory  hdfs://ns/data4/hadooptmp/spark-history
spark.eventLog.enabled  true
spark.eventLog.compress  true
spark.eventLog.dir  hdfs://ns/data4/hadooptmp/spark-history
spark.yarn.historyServer.address  172.31.10.119:18080
spark.io.compression.codec snappy
spark.executor.logs.rolling.enableCompression true
spark.dynamicAllocation.executorIdleTimeout 10s
spark.network.timeout 600s
spark.sql.parquet.writeLegacyFormat true