Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8626

Reserved RBW space is not released if creation of RBW File fails

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.1, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The DataNode reserves disk space for a full block when creating an RBW block and will release the space when the block is finalized (introduced in HDFS-6898)

      But if the RBW file creation fails, the reserved space is not released back.

      In a scenario, when the data node disk is full it causes no space left IOException. Eventually even if the disk got cleaned, the reserved space is not release until the Data Node is restarted

      Stacktrace for block creation failure is:

       IOException in BlockReceiver constructor. Cause is  | org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:264)
      java.io.IOException: No space left on device
      	at java.io.UnixFileSystem.createFileExclusively(Native Method)
      	at java.io.File.createNewFile(File.java:1012)
      	at org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
      	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:274)
      	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:762)
      	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1320)
      	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:190)
      	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:678)
      	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
      	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
      	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:250)
      	at java.lang.Thread.run(Thread.java:745)
      

        Attachments

        1. HDFS-8626-01.patch
          1 kB
          Kanaka Kumar Avvaru
        2. HDFS-8626-02.patch
          1 kB
          Kanaka Kumar Avvaru
        3. HDFS-8626-03.patch
          4 kB
          Kanaka Kumar Avvaru
        4. HDFS-8626-04_testcase_on_2.7.patch
          3 kB
          Kanaka Kumar Avvaru
        5. HDFS-8626-04.patch
          4 kB
          Kanaka Kumar Avvaru

          Issue Links

            Activity

              People

              • Assignee:
                kanaka Kanaka Kumar Avvaru
                Reporter:
                kanaka Kanaka Kumar Avvaru
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: