Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8626

Reserved RBW space is not released if creation of RBW File fails

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.1, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The DataNode reserves disk space for a full block when creating an RBW block and will release the space when the block is finalized (introduced in HDFS-6898)

      But if the RBW file creation fails, the reserved space is not released back.

      In a scenario, when the data node disk is full it causes no space left IOException. Eventually even if the disk got cleaned, the reserved space is not release until the Data Node is restarted

      Stacktrace for block creation failure is:

       IOException in BlockReceiver constructor. Cause is  | org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:264)
      java.io.IOException: No space left on device
      	at java.io.UnixFileSystem.createFileExclusively(Native Method)
      	at java.io.File.createNewFile(File.java:1012)
      	at org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
      	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:274)
      	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:762)
      	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1320)
      	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:190)
      	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:678)
      	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
      	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
      	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:250)
      	at java.lang.Thread.run(Thread.java:745)
      
      1. HDFS-8626-01.patch
        1 kB
        Kanaka Kumar Avvaru
      2. HDFS-8626-02.patch
        1 kB
        Kanaka Kumar Avvaru
      3. HDFS-8626-03.patch
        4 kB
        Kanaka Kumar Avvaru
      4. HDFS-8626-04.patch
        4 kB
        Kanaka Kumar Avvaru
      5. HDFS-8626-04_testcase_on_2.7.patch
        3 kB
        Kanaka Kumar Avvaru

        Issue Links

          Activity

          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for the updated test case Kanaka Kumar Avvaru. I have committed it to branch-2.7.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for the updated test case Kanaka Kumar Avvaru . I have committed it to branch-2.7.
          Hide
          kanaka Kanaka Kumar Avvaru added a comment -

          Attached patch on 2.7 branch only for the test case. Arpit Agarwal, please review and commit.

          Show
          kanaka Kanaka Kumar Avvaru added a comment - Attached patch on 2.7 branch only for the test case. Arpit Agarwal , please review and commit.
          Hide
          kanaka Kanaka Kumar Avvaru added a comment -

          Thanks Arpit Agarwal, for reviews and commit. Right now I don't have compiled code base for 2.7.. I will upload the test patch on 2.7 some time tomorrow.

          Show
          kanaka Kanaka Kumar Avvaru added a comment - Thanks Arpit Agarwal , for reviews and commit. Right now I don't have compiled code base for 2.7.. I will upload the test patch on 2.7 some time tomorrow.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2180 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2180/)
          HDFS-8626. Reserved RBW space is not released if creation of RBW File fails. (Contributed by kanaka kumar avvaru) (arp: rev 55278c012fbd5dcabee0276f8f42d0ab920cec93)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2180 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2180/ ) HDFS-8626 . Reserved RBW space is not released if creation of RBW File fails. (Contributed by kanaka kumar avvaru) (arp: rev 55278c012fbd5dcabee0276f8f42d0ab920cec93) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #232 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/232/)
          HDFS-8626. Reserved RBW space is not released if creation of RBW File fails. (Contributed by kanaka kumar avvaru) (arp: rev 55278c012fbd5dcabee0276f8f42d0ab920cec93)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #232 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/232/ ) HDFS-8626 . Reserved RBW space is not released if creation of RBW File fails. (Contributed by kanaka kumar avvaru) (arp: rev 55278c012fbd5dcabee0276f8f42d0ab920cec93) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #223 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/223/)
          HDFS-8626. Reserved RBW space is not released if creation of RBW File fails. (Contributed by kanaka kumar avvaru) (arp: rev 55278c012fbd5dcabee0276f8f42d0ab920cec93)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #223 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/223/ ) HDFS-8626 . Reserved RBW space is not released if creation of RBW File fails. (Contributed by kanaka kumar avvaru) (arp: rev 55278c012fbd5dcabee0276f8f42d0ab920cec93) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2162 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2162/)
          HDFS-8626. Reserved RBW space is not released if creation of RBW File fails. (Contributed by kanaka kumar avvaru) (arp: rev 55278c012fbd5dcabee0276f8f42d0ab920cec93)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2162 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2162/ ) HDFS-8626 . Reserved RBW space is not released if creation of RBW File fails. (Contributed by kanaka kumar avvaru) (arp: rev 55278c012fbd5dcabee0276f8f42d0ab920cec93) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #234 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/234/)
          HDFS-8626. Reserved RBW space is not released if creation of RBW File fails. (Contributed by kanaka kumar avvaru) (arp: rev 55278c012fbd5dcabee0276f8f42d0ab920cec93)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #234 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/234/ ) HDFS-8626 . Reserved RBW space is not released if creation of RBW File fails. (Contributed by kanaka kumar avvaru) (arp: rev 55278c012fbd5dcabee0276f8f42d0ab920cec93) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #964 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/964/)
          HDFS-8626. Reserved RBW space is not released if creation of RBW File fails. (Contributed by kanaka kumar avvaru) (arp: rev 55278c012fbd5dcabee0276f8f42d0ab920cec93)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #964 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/964/ ) HDFS-8626 . Reserved RBW space is not released if creation of RBW File fails. (Contributed by kanaka kumar avvaru) (arp: rev 55278c012fbd5dcabee0276f8f42d0ab920cec93) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8039 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8039/)
          HDFS-8626. Reserved RBW space is not released if creation of RBW File fails. (Contributed by kanaka kumar avvaru) (arp: rev 55278c012fbd5dcabee0276f8f42d0ab920cec93)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8039 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8039/ ) HDFS-8626 . Reserved RBW space is not released if creation of RBW File fails. (Contributed by kanaka kumar avvaru) (arp: rev 55278c012fbd5dcabee0276f8f42d0ab920cec93) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Committed for 2.7.1. Thanks Kanaka Kumar Avvaru.

          I did not include the test case in branch-2.7 as it causes a build failure. If you want to submit a separate patch with just the test for branch-2.7 I'd be happy to commit it, but it's fine as long as the test case is in branch-2.

          Show
          arpitagarwal Arpit Agarwal added a comment - Committed for 2.7.1. Thanks Kanaka Kumar Avvaru . I did not include the test case in branch-2.7 as it causes a build failure. If you want to submit a separate patch with just the test for branch-2.7 I'd be happy to commit it, but it's fine as long as the test case is in branch-2.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 57s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 33s There were no new javac warning messages.
          +1 javadoc 9m 42s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 16s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 3m 17s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 14s Pre-build of native portion
          +1 hdfs tests 162m 16s Tests passed in hadoop-hdfs.
              208m 46s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12740677/HDFS-8626-04.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / b42f1ec
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11413/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11413/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11413/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 57s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 33s There were no new javac warning messages. +1 javadoc 9m 42s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 16s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 3m 17s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 14s Pre-build of native portion +1 hdfs tests 162m 16s Tests passed in hadoop-hdfs.     208m 46s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12740677/HDFS-8626-04.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / b42f1ec hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11413/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11413/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11413/console This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thank you for fixing the test case Kanaka Kumar Avvaru! The v4 patch and test look great.

          +1 pending Jenkins.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thank you for fixing the test case Kanaka Kumar Avvaru ! The v4 patch and test look great. +1 pending Jenkins.
          Hide
          kanaka Kanaka Kumar Avvaru added a comment -

          Reflection was working fine Arpit Agarwal on 0th DN, but file creation was not failed may be because of multiple DNs in the MiniDFSCluster.
          Anyway to avoid confusion, I have updated patch to start single DN and expecting exception during file creation.

          Seems some problem with jenkins due to maintenance shutdown. Will check the failures if any in the next build.

          Show
          kanaka Kanaka Kumar Avvaru added a comment - Reflection was working fine Arpit Agarwal on 0th DN, but file creation was not failed may be because of multiple DNs in the MiniDFSCluster. Anyway to avoid confusion, I have updated patch to start single DN and expecting exception during file creation. Seems some problem with jenkins due to maintenance shutdown. Will check the failures if any in the next build.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 25s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 45s There were no new javac warning messages.
          +1 javadoc 9m 57s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 19s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          -1 findbugs 2m 55s Post-patch findbugs hadoop-hdfs-project/hadoop-hdfs compilation is broken.
          +1 findbugs 2m 55s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 0m 43s Pre-build of native portion
          -1 hdfs tests 157m 2s Tests failed in hadoop-hdfs.
              201m 39s  



          Reason Tests
          Failed unit tests hadoop.fs.TestHdfsNativeCodeLoader



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12740486/HDFS-8626-03.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 5b5bb8d
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11409/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11409/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11409/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 25s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 45s There were no new javac warning messages. +1 javadoc 9m 57s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 19s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 2m 55s Post-patch findbugs hadoop-hdfs-project/hadoop-hdfs compilation is broken. +1 findbugs 2m 55s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 0m 43s Pre-build of native portion -1 hdfs tests 157m 2s Tests failed in hadoop-hdfs.     201m 39s   Reason Tests Failed unit tests hadoop.fs.TestHdfsNativeCodeLoader Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12740486/HDFS-8626-03.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 5b5bb8d hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11409/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11409/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11409/console This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Kanaka Kumar Avvaru, thank you for adding the test case. I don't think the reflection trick in the test is working, else we'd get an exception while writing the file correct?

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Kanaka Kumar Avvaru , thank you for adding the test case. I don't think the reflection trick in the test is working, else we'd get an exception while writing the file correct?
          Hide
          kanaka Kanaka Kumar Avvaru added a comment -

          Updated patch with test case. Arpit Agarwal, please review.

          Show
          kanaka Kanaka Kumar Avvaru added a comment - Updated patch with test case. Arpit Agarwal , please review.
          Hide
          kanaka Kanaka Kumar Avvaru added a comment -

          I am trying to find a way to mock BlockPoolSlice/java.io.File to reproduce it in junit test case. But couldn't do it successfully yet.
          Down to File level can't be done as implementation looks different from java 7 & 8.

          If its urgent for 2.7.1, we can go a head,, Otherwise I will try for some time to add test case.

          Show
          kanaka Kanaka Kumar Avvaru added a comment - I am trying to find a way to mock BlockPoolSlice/java.io.File to reproduce it in junit test case. But couldn't do it successfully yet. Down to File level can't be done as implementation looks different from java 7 & 8. If its urgent for 2.7.1, we can go a head,, Otherwise I will try for some time to add test case.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for updating the patch.

          +1 for the change. Is it straightforward to add a test case?

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for updating the patch. +1 for the change. Is it straightforward to add a test case?
          Hide
          kanaka Kanaka Kumar Avvaru added a comment -

          jenkins gave result while I was writing comment. I have updated the patch for comment now.

          Show
          kanaka Kanaka Kumar Avvaru added a comment - jenkins gave result while I was writing comment. I have updated the patch for comment now.
          Hide
          kanaka Kanaka Kumar Avvaru added a comment -

          Thanks for the review Arpit Agarwal, I am aware of HDFS-8072 which is handling when client terminate while DN receiving the block.

          However this issue happens during the construction of block receiver as per the stack I observed. (Right now I don't have stack.. will update description with exception stack trace some time tomorrow)

          I will wait for the jenkins result before updating the patch for the given review comment.

          Show
          kanaka Kanaka Kumar Avvaru added a comment - Thanks for the review Arpit Agarwal , I am aware of HDFS-8072 which is handling when client terminate while DN receiving the block. However this issue happens during the construction of block receiver as per the stack I observed. (Right now I don't have stack.. will update description with exception stack trace some time tomorrow) I will wait for the jenkins result before updating the patch for the given review comment.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Canceling patch for addressing the comment above..

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Canceling patch for addressing the comment above..
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 48s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 31s There were no new javac warning messages.
          +1 javadoc 9m 38s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 11s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 3m 15s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 14s Pre-build of native portion
          +1 hdfs tests 162m 30s Tests passed in hadoop-hdfs.
              208m 41s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12740370/HDFS-8626-01.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 2ad6687
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11403/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11403/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11403/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 48s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 31s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 11s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 3m 15s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 14s Pre-build of native portion +1 hdfs tests 162m 30s Tests passed in hadoop-hdfs.     208m 41s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12740370/HDFS-8626-01.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 2ad6687 hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11403/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11403/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11403/console This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Kanaka Kumar Avvaru Nice finding...And thanks for working on this..+1 (non-binding ) after addressing Arpit Agarwal comment.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Kanaka Kumar Avvaru Nice finding...And thanks for working on this..+1 (non-binding ) after addressing Arpit Agarwal comment.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          AFAIK, it is different from HDFS-8072..

          Show
          brahmareddy Brahma Reddy Battula added a comment - AFAIK, it is different from HDFS-8072 ..
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          The patch looks good. Nitpick - you can simplify the try block.

              try {
                return getBlockPoolSlice(bpid).createRbwFile(b);
              } catch (IOException exception) {
          

          +1 with that fixed.

          Show
          arpitagarwal Arpit Agarwal added a comment - The patch looks good. Nitpick - you can simplify the try block. try { return getBlockPoolSlice(bpid).createRbwFile(b); } catch (IOException exception) { +1 with that fixed.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Kanaka Kumar Avvaru, have you ruled out HDFS-8072? I'll take a look at this patch.

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Kanaka Kumar Avvaru , have you ruled out HDFS-8072 ? I'll take a look at this patch.
          Hide
          kanaka Kanaka Kumar Avvaru added a comment -

          Thanks Kihwal Lee, In that scenario we had only one disk path configured for data node.
          Also for information, get availability on separate Java code test and df command gives available space from disk as 100+ GB but data node jmx report shown as 3 GB after few hours also.

          In order to confirm the leak , I have ensured through datanode heap dump analysis that reservedRBW variable in FSVolumeImpl is holding the number equalent to missing 100+ GB.

          Show
          kanaka Kanaka Kumar Avvaru added a comment - Thanks Kihwal Lee , In that scenario we had only one disk path configured for data node. Also for information, get availability on separate Java code test and df command gives available space from disk as 100+ GB but data node jmx report shown as 3 GB after few hours also. In order to confirm the leak , I have ensured through datanode heap dump analysis that reservedRBW variable in FSVolumeImpl is holding the number equalent to missing 100+ GB.
          Hide
          kihwal Kihwal Lee added a comment -

          In a scenario, when the data node disk is full it causes no space left IOException.

          If this was truely the cause, it indicates getNextVolume() is not working properly. Normally DiskOutOfSpaceException is thrown before any space is reserved. So, if you believe it was caused by ENOSPACE, try finding out why the remaining space check was not working. We have seen bad disks causing ext4 to return incorrect values to the du command. In that case, the getAvailable() may not return correct result.

          In any case, the proposed change is still good for any other types of file creation failures. Arpit Agarwal, what do you think?

          Show
          kihwal Kihwal Lee added a comment - In a scenario, when the data node disk is full it causes no space left IOException. If this was truely the cause, it indicates getNextVolume() is not working properly. Normally DiskOutOfSpaceException is thrown before any space is reserved. So, if you believe it was caused by ENOSPACE , try finding out why the remaining space check was not working. We have seen bad disks causing ext4 to return incorrect values to the du command. In that case, the getAvailable() may not return correct result. In any case, the proposed change is still good for any other types of file creation failures. Arpit Agarwal , what do you think?
          Hide
          kanaka Kanaka Kumar Avvaru added a comment -

          Attached patch to release reserved space in case of IOException while creating the RBW file

          Show
          kanaka Kanaka Kumar Avvaru added a comment - Attached patch to release reserved space in case of IOException while creating the RBW file
          Hide
          kanaka Kanaka Kumar Avvaru added a comment -

          Right now I don't have unit test case to reproduce but tried stubbing DatanodeUtil.createTmpFile(...) to throw IOException unconditionally.

          Show
          kanaka Kanaka Kumar Avvaru added a comment - Right now I don't have unit test case to reproduce but tried stubbing DatanodeUtil.createTmpFile(...) to throw IOException unconditionally.

            People

            • Assignee:
              kanaka Kanaka Kumar Avvaru
              Reporter:
              kanaka Kanaka Kumar Avvaru
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development