-
Type:
Bug
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 3.3.0
-
Component/s: None
-
Labels:None
-
Release Note:Reduce the output stream buffer size of a DFSClient remote read from 8KB to 512 bytes.
this issue is similar to HDFS-14535.
public static BlockReader newBlockReader(String file, ExtendedBlock block, Token<BlockTokenIdentifier> blockToken, long startOffset, long len, boolean verifyChecksum, String clientName, Peer peer, DatanodeID datanodeID, PeerCache peerCache, CachingStrategy cachingStrategy, int networkDistance) throws IOException { // in and out will be closed when sock is closed (by the caller) final DataOutputStream out = new DataOutputStream(new BufferedOutputStream( peer.getOutputStream())); new Sender(out).readBlock(block, blockToken, clientName, startOffset, len, verifyChecksum, cachingStrategy); } public BufferedOutputStream(OutputStream out) { this(out, 8192); }
Sender#readBlock parameter( block,blockToken, clientName, startOffset, len, verifyChecksum, cachingStrategy) could not use such a big buffer.
So i think it should reduce BufferedOutputStream buffer.
- is related to
-
HDFS-14844 Make buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream configurable
-
- Resolved
-
- relates to
-
HDFS-14535 The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read
-
- Resolved
-