[HDFS-14294] Deprecate DFSUtilClient#getSmallBufferSize

• Type: Improvement
• Status: Patch Available
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.2.0
• Component/s: hdfs
• Assignee: David Mollitor
• Reporter: David Mollitor
• Created: 18/Feb/19 14:53
• Updated: 21/Feb/19 19:59

Description

DFSUtilClient.java

  public static int getIoFileBufferSize(Configuration conf) {
    return conf.getInt(
        CommonConfigurationKeysPublic.IO_FILE_BUFFER_SIZE_KEY,
        CommonConfigurationKeysPublic.IO_FILE_BUFFER_SIZE_DEFAULT);
  }

 public static int getSmallBufferSize(Configuration conf) {
    return Math.min(getIoFileBufferSize(conf) / 2, 512);
  }

This concept of a "small buffer size" seems a bit overkill. First of all, it's not documented that such a thing exists and that by adjusting dfs.stream-buffer-size an administrator is also scaling these other buffer sizes. Seconds, I think any "small" buffer size should just use the default JDK buffer sizes. Anything that benefits from being larger than the default JDK size should be the controlled by IO_FILE_BUFFER_SIZE_KEY / dfs.stream-buffer-size. For reference, the default JDK size is 8K : HDFS-14293

Issue Links

• is related to: HDFS-14293 Increase Default Size of dfs.stream-buffer-size