[SPARK-34346] io.file.buffer.size set by spark.buffer.size will override by hive-site.xml may cause perf regression

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.0.2, 3.1.1
• Fix Version/s: 3.0.2, 3.1.1
• Component/s: Spark Core, SQL
• Assignee: Kent Yao 2
• Reporter: Kent Yao 2
• Created: 03/Feb/21 14:13
• Updated: 13/Feb/21 11:07
• Resolved: 05/Feb/21 21:21

Description

In many real-world cases, when interacting with hive catalog through Spark SQL, users may just share the `hive-site.xml` for their hive jobs and make a copy to `SPARK_HOME`/conf w/o modification. In Spark, when we generate Hadoop configurations, we will use `spark.buffer.size(65536)` to reset `io.file.buffer.size(4096)`. But when we load the hive-site.xml, we may ignore this behavior and reset `io.file.buffer.size` again according to `hive-site.xml`.

1. The configuration priority for setting Hadoop and Hive config here is not right, while literally, the order should be `spark > spark.hive > spark.hadoop > hive > hadoop`

2. This breaks `spark.buffer.size` congfig's behavior for tuning the IO performance w/ HDFS if there is an existing `io.file.buffer.size` in hive-site.xml

Issue Links

• is related to: SPARK-34431 Only load hive-site.xml once
• links to: [Github] Pull Request #31460 (yaooqinn)
• links to: [Github] Pull Request #31460 (yaooqinn)
• links to: [Github] Pull Request #31482 (yaooqinn)
• links to: [Github] Pull Request #31492 (yaooqinn)
• links to: [Github] Pull Request #31492 (yaooqinn)
• links to: [Github] Pull Request #31515 (dongjoon-hyun)