[SPARK-25356] Add Parquet block size (row group size) option to SparkSQL configuration

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Invalid
• Affects Version/s: 2.4.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: liuxian
• Created: 06/Sep/18 10:37
• Updated: 07/Sep/18 01:03
• Resolved: 07/Sep/18 01:03

Description

I think we should configure the Parquet buffer size when using Parquet format.

Because for HDFS, `dfs.block.size` is configurable, sometimes we hope the block size of parquet to be consistent with it.

And  whether this parameter `spark.sql.files.maxPartitionBytes` is best consistent with the Parquet  block size when using Parquet format?

Issue Links

• links to: [Github] Pull Request #22350 (10110346)