-
Type:
Improvement
-
Status: Patch Available
-
Priority:
Minor
-
Resolution: Unresolved
-
Affects Version/s: 3.2.0
-
Fix Version/s: None
-
Component/s: hdfs
-
Labels:None
public static int getIoFileBufferSize(Configuration conf) { return conf.getInt( CommonConfigurationKeysPublic.IO_FILE_BUFFER_SIZE_KEY, CommonConfigurationKeysPublic.IO_FILE_BUFFER_SIZE_DEFAULT); } public static int getSmallBufferSize(Configuration conf) { return Math.min(getIoFileBufferSize(conf) / 2, 512); }
This concept of a "small buffer size" seems a bit overkill. First of all, it's not documented that such a thing exists and that by adjusting dfs.stream-buffer-size an administrator is also scaling these other buffer sizes. Seconds, I think any "small" buffer size should just use the default JDK buffer sizes. Anything that benefits from being larger than the default JDK size should be the controlled by IO_FILE_BUFFER_SIZE_KEY / dfs.stream-buffer-size. For reference, the default JDK size is 8K : HDFS-14293
- is related to
-
HDFS-14293 Increase Default Size of dfs.stream-buffer-size
-
- Open
-