Details
-
Improvement
-
Status: Patch Available
-
Minor
-
Resolution: Unresolved
-
3.2.0
-
None
-
None
Description
DFSUtilClient.java
public static int getIoFileBufferSize(Configuration conf) { return conf.getInt( CommonConfigurationKeysPublic.IO_FILE_BUFFER_SIZE_KEY, CommonConfigurationKeysPublic.IO_FILE_BUFFER_SIZE_DEFAULT); } public static int getSmallBufferSize(Configuration conf) { return Math.min(getIoFileBufferSize(conf) / 2, 512); }
This concept of a "small buffer size" seems a bit overkill. First of all, it's not documented that such a thing exists and that by adjusting dfs.stream-buffer-size an administrator is also scaling these other buffer sizes. Seconds, I think any "small" buffer size should just use the default JDK buffer sizes. Anything that benefits from being larger than the default JDK size should be the controlled by IO_FILE_BUFFER_SIZE_KEY / dfs.stream-buffer-size. For reference, the default JDK size is 8K : HDFS-14293
Attachments
Attachments
Issue Links
- is related to
-
HDFS-14293 Increase Default Size of dfs.stream-buffer-size
- Open