[SPARK-18432] Fix HDFS block size in programming guide

• Type: Documentation
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 2.0.1
• Fix Version/s: 2.0.3, 2.1.0
• Component/s: Documentation
• Assignee: Noritaka Sekiyama
• Reporter: Noritaka Sekiyama
• Created: 14/Nov/16 07:31
• Updated: 14/Nov/16 12:21
• Resolved: 14/Nov/16 12:11

Description

http://spark.apache.org/docs/latest/programming-guide.html
"By default, Spark creates one partition for each block of the file (blocks being 64MB by default in HDFS)"

Currently default block size in HDFS is 128MB.
The default value has been already increased in Hadoop 2.2.0 (the oldest supported version of Spark). https://issues.apache.org/jira/browse/HDFS-4053

Since it looks confusing explanation, I'd like to fix the value from 64MB to 128MB.

Issue Links

• links to: [Github] Pull Request #15879 (moomindani)