[HADOOP-212] allow changes to dfs block size

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.2.0
• Fix Version/s: 0.3.0
• Assignee: Owen O'Malley
• Reporter: Owen O'Malley
• Created: 12/May/06 22:45
• Updated: 08/Jul/09 16:41
• Resolved: 16/May/06 04:26

Description

Trying to change the DFS block size, led the realization that the 32,000,000 was hard coded into the source code. I propose:
1. Change the default block size to 64 * 1024 * 1024.
2. Add the config variable dfs.block.size that sets the default block size.
3. Add a parameter to the FileSystem, DFSClient, and ClientProtocol create method that let's the user control the block size.
4. Rename the FileSystem.getBlockSize to getDefaultBlockSize.
5. Add a new method to FileSytem.getBlockSize that takes a pathname.
6. Use long for the block size in the API, which is what was used before. However, the implementation will not work if block size is set bigger than 2**31.
7. Have the InputFormatBase use the blocksize of each file to determine the split size.

Thoughts?