[HDFS-6758] block writer should pass the expected block size to DataXceiverServer

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.1
• Fix Version/s: 2.6.0
• Component/s: datanode, hdfs-client
• Target Version/s: 2.6.0
• Hadoop Flags: Reviewed
• Assignee: Arpit Agarwal
• Reporter: Arpit Agarwal
• Created: 25/Jul/14 22:56
• Updated: 01/Dec/14 03:10
• Resolved: 21/Aug/14 01:14

Description

DataXceiver initializes the block size to the default block size for the cluster. This size is later used by the FsDatasetImpl when applying VolumeChoosingPolicy.

    block.setNumBytes(dataXceiverServer.estimateBlockSize);

where

  /**
   * We need an estimate for block size to check if the disk partition has
   * enough space. For now we set it to be the default block size set
   * in the server side configuration, which is not ideal because the
   * default block size should be a client-size configuration. 
   * A better solution is to include in the header the estimated block size,
   * i.e. either the actual block size or the default block size.
   */
  final long estimateBlockSize;

In most cases the writer can just pass the maximum expected block size to the DN instead of having to use the cluster default.

Issue Links

• is required by: HDFS-6898 DN must reserve space for a full block when an RBW block is created