[HDFS-13642] Creating a file with block size smaller than EC policy's cell size should fail

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.2.0, 3.1.3
• Component/s: erasure-coding
• Target Version/s: 3.2.0
• Hadoop Flags: Reviewed
• Assignee: Xiao Chen
• Reporter: Xiao Chen
• Created: 30/May/18 18:07
• Updated: 23/Aug/19 06:08
• Resolved: 08/Jun/18 22:22

Description

The following command causes an exception:

hadoop fs -Ddfs.block.size=349696 -put -f lineitem_sixblocks.parquet /test-warehouse/tmp123ec

18/05/25 16:00:59 WARN hdfs.DataStreamer: DataStreamer Exception
java.io.IOException: BlockSize 349696 < lastByteOffsetInBlock, #0: blk_-9223372036854574256_14634, packet seqno: 7 offsetInBlock: 349696 lastPacketInBlock: false lastByteOffsetInBlock: 350208
  at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:729)
  at org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:46)
18/05/25 16:00:59 WARN hdfs.DFSOutputStream: Failed: offset=4096, length=512, DFSStripedOutputStream:#0: failed, blk_-9223372036854574256_14634
java.io.IOException: BlockSize 349696 < lastByteOffsetInBlock, #0: blk_-9223372036854574256_14634, packet seqno: 7 offsetInBlock: 349696 lastPacketInBlock: false lastByteOffsetInBlock: 350208
  at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:729)
  at org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:46)

Then the streamer is confused and hangs.

The local file is under 6MB, the hdfs file has a RS-3-2-1024k EC policy.

 

Credit to tarasbob for reporting this issue.