Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13642

Creating a file with block size smaller than EC policy's cell size should fail

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.2.0
    • Component/s: erasure-coding
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The following command causes an exception:

      hadoop fs -Ddfs.block.size=349696 -put -f lineitem_sixblocks.parquet /test-warehouse/tmp123ec
      
      18/05/25 16:00:59 WARN hdfs.DataStreamer: DataStreamer Exception
      java.io.IOException: BlockSize 349696 < lastByteOffsetInBlock, #0: blk_-9223372036854574256_14634, packet seqno: 7 offsetInBlock: 349696 lastPacketInBlock: false lastByteOffsetInBlock: 350208
        at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:729)
        at org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:46)
      18/05/25 16:00:59 WARN hdfs.DFSOutputStream: Failed: offset=4096, length=512, DFSStripedOutputStream:#0: failed, blk_-9223372036854574256_14634
      java.io.IOException: BlockSize 349696 < lastByteOffsetInBlock, #0: blk_-9223372036854574256_14634, packet seqno: 7 offsetInBlock: 349696 lastPacketInBlock: false lastByteOffsetInBlock: 350208
        at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:729)
        at org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:46)
      

      Then the streamer is confused and hangs.

      The local file is under 6MB, the hdfs file has a RS-3-2-1024k EC policy.

       

      Credit to Taras Bobrovytsky for reporting this issue.

        Attachments

        1. HDFS-13642.01.patch
          6 kB
          Xiao Chen
        2. editsStored
          8 kB
          Xiao Chen
        3. HDFS-13642.02.patch
          10 kB
          Xiao Chen
        4. HDFS-13642.03.patch
          10 kB
          Xiao Chen

          Activity

            People

            • Assignee:
              xiaochen Xiao Chen
              Reporter:
              xiaochen Xiao Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: