[HDFS-14820] The default 8KB buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream is too big

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Release Note: Update the output stream buffer size of a DFSClient remote read.
• Assignee: Lisheng Sun
• Reporter: Lisheng Sun
• Created: 04/Sep/19 11:58
• Updated: 28/Nov/19 15:24

Description

this issue is similar to HDFS-14535.

public static BlockReader newBlockReader(String file,
    ExtendedBlock block,
    Token<BlockTokenIdentifier> blockToken,
    long startOffset, long len,
    boolean verifyChecksum,
    String clientName,
    Peer peer, DatanodeID datanodeID,
    PeerCache peerCache,
    CachingStrategy cachingStrategy,
    int networkDistance) throws IOException {
  // in and out will be closed when sock is closed (by the caller)
  final DataOutputStream out = new DataOutputStream(new BufferedOutputStream(
      peer.getOutputStream()));
  new Sender(out).readBlock(block, blockToken, clientName, startOffset, len,
      verifyChecksum, cachingStrategy);
}

public BufferedOutputStream(OutputStream out) {
    this(out, 8192);
}

Sender#readBlock parameter( block,blockToken, clientName, startOffset, len, verifyChecksum, cachingStrategy) could not use such a big buffer.
So i think it should reduce BufferedOutputStream buffer.

Issue Links

• is related to: HDFS-14844 Make buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream configurable
• relates to: HDFS-14535 The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read