Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-5215

dfs.datanode.du.reserved is not considered while computing available space

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.0, 3.0.0-alpha1
    • Fix Version/s: 2.8.0, 2.7.1, 3.0.0-alpha1
    • Component/s: datanode
    • Labels:
      None

      Description

      public long getAvailable() throws IOException {
          long remaining = getCapacity()-getDfsUsed();
          long available = usage.getAvailable();
          if (remaining > available) {
            remaining = available;
          }
          return (remaining > 0) ? remaining : 0;
        } 
      

      Here we are not considering the reserved space while getting the Available Space.

      1. HDFS-5215.patch
        0.8 kB
        Brahma Reddy Battula
      2. HDFS-5215-002.patch
        0.9 kB
        Brahma Reddy Battula
      3. HDFS-5215-003.patch
        2 kB
        Brahma Reddy Battula
      4. HDFS-5215-004.patch
        2 kB
        Brahma Reddy Battula
      5. HDFS-5215-005.patch
        3 kB
        Brahma Reddy Battula

        Issue Links

          Activity

          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Fix can be

          long available = usage.getAvailable() - reserved;

          Please let me know any comments on this..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Fix can be long available = usage.getAvailable() - reserved; Please let me know any comments on this..
          Hide
          umamaheswararao Uma Maheswara Rao G added a comment -

          I think its a nice finding Brahma, planning to post a patch?

          You can also add a test which can detect this issue. We may not find this issue until you have some unuseable space in the partition right?

          Show
          umamaheswararao Uma Maheswara Rao G added a comment - I think its a nice finding Brahma, planning to post a patch? You can also add a test which can detect this issue. We may not find this issue until you have some unuseable space in the partition right?
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Uma Mahesh Thanks for taking a look at this issue..

          Will post the patch with testcase.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Uma Mahesh Thanks for taking a look at this issue.. Will post the patch with testcase.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Tested manually, after defect fix..it's working fine

          Show
          brahmareddy Brahma Reddy Battula added a comment - Tested manually, after defect fix..it's working fine
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12610538/HDFS-5215.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5291//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5291//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12610538/HDFS-5215.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5291//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5291//console This message is automatically generated.
          Hide
          sureshms Suresh Srinivas added a comment -

          Here is the branch-2 code:

            /**
             * Calculate the capacity of the filesystem, after removing any
             * reserved capacity.
             * @return the unreserved number of bytes left in this filesystem. May be zero.
             */
            long getCapacity() {
              long remaining = usage.getCapacity() - reserved;
              return remaining > 0 ? remaining : 0;
            }
          
            @Override
            public long getAvailable() throws IOException {
              long remaining = getCapacity()-getDfsUsed();
              long available = usage.getAvailable();
              if (remaining > available) {
                remaining = available;
              }
              return (remaining > 0) ? remaining : 0;
            }
          

          The logic seems to be correct.

          • remaining is total capacity - reserved - dfs used. That is the space available for DFS use without considering actual space available on the disk.
          • available is space currently available.

          The remaining space returned is smaller of the above two. This seems correct to me. What is the need to remove reserved space from available above?

          Show
          sureshms Suresh Srinivas added a comment - Here is the branch-2 code: /** * Calculate the capacity of the filesystem, after removing any * reserved capacity. * @ return the unreserved number of bytes left in this filesystem. May be zero. */ long getCapacity() { long remaining = usage.getCapacity() - reserved; return remaining > 0 ? remaining : 0; } @Override public long getAvailable() throws IOException { long remaining = getCapacity()-getDfsUsed(); long available = usage.getAvailable(); if (remaining > available) { remaining = available; } return (remaining > 0) ? remaining : 0; } The logic seems to be correct. remaining is total capacity - reserved - dfs used. That is the space available for DFS use without considering actual space available on the disk. available is space currently available. The remaining space returned is smaller of the above two. This seems correct to me. What is the need to remove reserved space from available above?
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          The remaining space returned is smaller of the above two. This seems correct to me. What is the need to remove reserved space from available above?

          AS we are assigning the available to remaining,there is one case where always remaining will be greater than available space.. this scenario will come where dfsused is very less but disk got filled with non-dfs data..

            if (remaining > available) {
                remaining = available;
              }
          

          Check following for same..

          TEST ENV: total space of datanode disk is 69G, revered space 10G, available space is 66G at beginning, dd command generate files use 52G space.
          Debug the old code:
          long remaining = getCapacity()-getDfsUsed(); – 63242600448
          long available = usage.getAvailable(); – 14144172032
          if (remaining > available)

          { remaining = available; -- 14144172032 }

          Upload 6 files, as per file has 666266906 bytes:
          long remaining = getCapacity()-getDfsUsed(); – 59213767320
          long available = usage.getAvailable(); – 10110709760
          if (remaining > available)

          { remaining = available; -- 10110709760 }

          Upload 7th file success.
          Clear the env and degug the new code:
          long remaining = getCapacity()-getDfsUsed(); – 63238742828
          long available = usage.getAvailable()- reserved; – 3406753792
          if (remaining > available)

          { remaining = available; -- 3406753792 }

          Upload 6 files, as per file has 666266906 bytes:
          long remaining = getCapacity()-getDfsUsed(); – 59881381888
          long available = usage.getAvailable()- reserved; – 45535232
          if (remaining > available)

          { remaining = available; -- 45535232 }

          Upload 7th file failed.

          Show
          brahmareddy Brahma Reddy Battula added a comment - The remaining space returned is smaller of the above two. This seems correct to me. What is the need to remove reserved space from available above? AS we are assigning the available to remaining,there is one case where always remaining will be greater than available space.. this scenario will come where dfsused is very less but disk got filled with non-dfs data.. if (remaining > available) { remaining = available; } Check following for same.. TEST ENV: total space of datanode disk is 69G, revered space 10G, available space is 66G at beginning, dd command generate files use 52G space. Debug the old code: long remaining = getCapacity()-getDfsUsed(); – 63242600448 long available = usage.getAvailable(); – 14144172032 if (remaining > available) { remaining = available; -- 14144172032 } Upload 6 files, as per file has 666266906 bytes: long remaining = getCapacity()-getDfsUsed(); – 59213767320 long available = usage.getAvailable(); – 10110709760 if (remaining > available) { remaining = available; -- 10110709760 } Upload 7th file success. Clear the env and degug the new code: long remaining = getCapacity()-getDfsUsed(); – 63238742828 long available = usage.getAvailable()- reserved; – 3406753792 if (remaining > available) { remaining = available; -- 3406753792 } Upload 6 files, as per file has 666266906 bytes: long remaining = getCapacity()-getDfsUsed(); – 59881381888 long available = usage.getAvailable()- reserved; – 45535232 if (remaining > available) { remaining = available; -- 45535232 } Upload 7th file failed.
          Hide
          fahlke Alexander Fahlke added a comment -

          Have you considered that the underlying filesystem reserved space (ext3, ext4 -> tune2fs) can also affect this behavior?
          I'm not sure if this is a problem in hadoop-3.0.x anymore.

          I recently opened a jira regarding this issue: https://issues.apache.org/jira/browse/HDFS-5926

          Show
          fahlke Alexander Fahlke added a comment - Have you considered that the underlying filesystem reserved space (ext3, ext4 -> tune2fs) can also affect this behavior? I'm not sure if this is a problem in hadoop-3.0.x anymore. I recently opened a jira regarding this issue: https://issues.apache.org/jira/browse/HDFS-5926
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12610538/HDFS-5215.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6587//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6587//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12610538/HDFS-5215.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6587//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6587//console This message is automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Hi Alexander Fahlke,
          Thanks for taking look into this issue..

          Have you considered that the underlying filesystem reserved space (ext3, ext4 -> tune2fs) can also affect this behavior?

          Yes, I checked in ext3 file system only..

          I'm not sure if this is a problem in hadoop-3.0.x anymore.

          As I knew , it is not fixed..problem will be present in hadoop-3.0.x . .Please let me know your thoughts on this...

          Show
          brahmareddy Brahma Reddy Battula added a comment - Hi Alexander Fahlke , Thanks for taking look into this issue.. Have you considered that the underlying filesystem reserved space (ext3, ext4 -> tune2fs) can also affect this behavior? Yes, I checked in ext3 file system only.. I'm not sure if this is a problem in hadoop-3.0.x anymore. As I knew , it is not fixed..problem will be present in hadoop-3.0.x . .Please let me know your thoughts on this...
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12610538/HDFS-5215.patch
          against trunk revision 14e2639.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8019//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12610538/HDFS-5215.patch against trunk revision 14e2639. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8019//console This message is automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Any thoughts on this jira..?

          Show
          brahmareddy Brahma Reddy Battula added a comment - Any thoughts on this jira..?
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12610538/HDFS-5215.patch
          against trunk revision ed70fa1.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9736//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12610538/HDFS-5215.patch against trunk revision ed70fa1. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9736//console This message is automatically generated.
          Hide
          yzhangal Yongjun Zhang added a comment -

          HI Brahma Reddy Battula, thanks for reporting the issue and the patch here. The patch doesn't apply anymore. would you upload a new rev? Thanks.

          HI Suresh Srinivas,

          Brahma's reply to your comment

          https://issues.apache.org/jira/browse/HDFS-5215?focusedCommentId=13820096&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13820096

          seems reasonable to me. Would you please comment?

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - HI Brahma Reddy Battula , thanks for reporting the issue and the patch here. The patch doesn't apply anymore. would you upload a new rev? Thanks. HI Suresh Srinivas , Brahma's reply to your comment https://issues.apache.org/jira/browse/HDFS-5215?focusedCommentId=13820096&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13820096 seems reasonable to me. Would you please comment? Thanks.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hm, seems it's better we deduct reserved from DF.java:

           public long getAvailable() {
              return dirFile.getUsableSpace();
            }
          

          That is, replace the above with {{ dirFile.getUsableSpace() - reserved; }}, if we define DF#getAvailable() as the space available to HDFS.

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Hm, seems it's better we deduct reserved from DF.java: public long getAvailable() { return dirFile.getUsableSpace(); } That is, replace the above with {{ dirFile.getUsableSpace() - reserved; }}, if we define DF#getAvailable() as the space available to HDFS. Thanks.
          Hide
          yzhangal Yongjun Zhang added a comment -

          DF#getAvailable() is called by LocalDirAllocator#getLocalPathForWrite and others, so I think the change I suggested in my last comment would make the patch more comprehensive. Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - DF#getAvailable() is called by LocalDirAllocator#getLocalPathForWrite and others, so I think the change I suggested in my last comment would make the patch more comprehensive. Thanks.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Thanks a lot for taking a look into this issue..Yes, I am waiting for Suresh Srinivas reply..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Thanks a lot for taking a look into this issue..Yes, I am waiting for Suresh Srinivas reply..
          Hide
          yzhangal Yongjun Zhang added a comment -

          Sure Brahma. The patch doesn't apply anymore. I also have some review comments down there, wonder if you could submit an updated patch, so people have a new base to discuss? thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Sure Brahma. The patch doesn't apply anymore. I also have some review comments down there, wonder if you could submit an updated patch, so people have a new base to discuss? thanks.
          Hide
          yzhangal Yongjun Zhang added a comment -

          HI Brahma Reddy Battula,

          I noticed that I did not ping your name when I did the following comment, I'm putting it here again, thanks for following up.

          The patch doesn't apply anymore. I also have some review comments, wonder if you could submit an updated patch, so people have a new base to discuss? thanks

          Show
          yzhangal Yongjun Zhang added a comment - HI Brahma Reddy Battula , I noticed that I did not ping your name when I did the following comment, I'm putting it here again, thanks for following up. The patch doesn't apply anymore. I also have some review comments, wonder if you could submit an updated patch, so people have a new base to discuss? thanks
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Brahma Reddy Battula,

          I studied the callers of DF#getAvailable() including LocalDirAllocator#getLocalPathForWrite a little more and realize that they don't need to be aware of the reserved space specified by dfs.datanode.du.reserved. So we don't need to touch the implementation in DF.java.

          Sorry for the forth and back.

          Show
          yzhangal Yongjun Zhang added a comment - Hi Brahma Reddy Battula , I studied the callers of DF#getAvailable() including LocalDirAllocator#getLocalPathForWrite a little more and realize that they don't need to be aware of the reserved space specified by dfs.datanode.du.reserved . So we don't need to touch the implementation in DF.java. Sorry for the forth and back.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12707741/HDFS-5215-002.patch
          against trunk revision af618f2.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The test build failed in hadoop-hdfs-project/hadoop-hdfs

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10088//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10088//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707741/HDFS-5215-002.patch against trunk revision af618f2. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The test build failed in hadoop-hdfs-project/hadoop-hdfs Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10088//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10088//console This message is automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Testcase failures are unrelated this patch..@Yongjun Zhang Kindly Review..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Testcase failures are unrelated this patch..@Yongjun Zhang Kindly Review..
          Hide
          yzhangal Yongjun Zhang added a comment -

          HI Brahma Reddy Battula,

          Thanks for updating the patch. It looks good to me, some comments:

          1. add javadoc for getAvailable(), like how getCapacity() method does.
          2. add comment to, the freeSpace is now excluding reserved after this patch:
            private static class VolumeInfo {
                final String directory;
                final long usedSpace;
                final long freeSpace;
                final long reservedSpace;
            
          3. if possible, can we add a test or modify TestNamenodeCapacityReport#testVolumeSize to see the impact of this change?

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - HI Brahma Reddy Battula , Thanks for updating the patch. It looks good to me, some comments: add javadoc for getAvailable() , like how getCapacity() method does. add comment to, the freeSpace is now excluding reserved after this patch: private static class VolumeInfo { final String directory; final long usedSpace; final long freeSpace; final long reservedSpace; if possible, can we add a test or modify TestNamenodeCapacityReport#testVolumeSize to see the impact of this change? Thanks.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Thanks a lot for review...Updated the patch based on your comments..Tested manually which I mentioned earlier in this jira.getraimining() will be called in TestNamenodeCapacityReport#testVolumeSize which will internally calls getavailable(), Hence I did not updated this.. Hopefully this should be ok..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Thanks a lot for review...Updated the patch based on your comments..Tested manually which I mentioned earlier in this jira. getraimining() will be called in TestNamenodeCapacityReport#testVolumeSize which will internally calls getavailable() , Hence I did not updated this.. Hopefully this should be ok..
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12708742/HDFS-5215-003.patch
          against trunk revision 4922394.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javac. The applied patch generated 1153 javac compiler warnings (more than the trunk's current 1151 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs:

          org.apache.hadoop.mapreduce.v2.app.TestMRClientService
          org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryParsing

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10143//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/10143//artifact/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10143//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12708742/HDFS-5215-003.patch against trunk revision 4922394. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac . The applied patch generated 1153 javac compiler warnings (more than the trunk's current 1151 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs: org.apache.hadoop.mapreduce.v2.app.TestMRClientService org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryParsing Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10143//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/10143//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10143//console This message is automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Testcase failures and javadoc wars are unrelated this patch.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Testcase failures and javadoc wars are unrelated this patch.
          Hide
          sinago zhouyingchao added a comment -

          Shouldn't we also subtract rbwReserved ?

          Show
          sinago zhouyingchao added a comment - Shouldn't we also subtract rbwReserved ?
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Thanks for Review,, I will update patch..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Thanks for Review,, I will update patch..
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Updated the patch...Yongjun Zhang kindly review...Thanks..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Updated the patch... Yongjun Zhang kindly review...Thanks..
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12709278/HDFS-5215-004.patch
          against trunk revision db80e42.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics

          The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10174//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10174//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12709278/HDFS-5215-004.patch against trunk revision db80e42. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10174//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10174//console This message is automatically generated.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Brahma Reddy Battula,

          Sorry for the delay, I was buried in other stuff.

          I agree with zhouyingchao that including rbwReserved is reasonable. Suggest to change the comment

          1.
          remove {{* the freeSpace is now excluding reserved + rbw after HDFS-5215 }} and add comments like blow

             final long usedSpace;     // size of space used by HDFS
             final long freeSpace;     // size of free space excluding reserved space
             final long reservedSpace; // size of space reserved for non-HDFS and RBW
          

          2.

           /**
             * Return either the configured capacity of the file system if configured;
             * or the capacity of the file system excluding space reserved for non-HDFS.
             * @return the unreserved number of bytes left in this filesystem. May be zero.
             */
            @VisibleForTesting
            public long getCapacity() {
          

          3.

           /*
             * Calculate the available space of the filesystem, excluding space reserved
             * for non-HDFS and space reserved for RBW
             * 
             * @return the available number of bytes left in this filesystem. May be zero.
             */
            @Override
            public long getAvailable() throws IOException {
          

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Hi Brahma Reddy Battula , Sorry for the delay, I was buried in other stuff. I agree with zhouyingchao that including rbwReserved is reasonable. Suggest to change the comment 1. remove {{* the freeSpace is now excluding reserved + rbw after HDFS-5215 }} and add comments like blow final long usedSpace; // size of space used by HDFS final long freeSpace; // size of free space excluding reserved space final long reservedSpace; // size of space reserved for non-HDFS and RBW 2. /** * Return either the configured capacity of the file system if configured; * or the capacity of the file system excluding space reserved for non-HDFS. * @ return the unreserved number of bytes left in this filesystem. May be zero. */ @VisibleForTesting public long getCapacity() { 3. /* * Calculate the available space of the filesystem, excluding space reserved * for non-HDFS and space reserved for RBW * * @ return the available number of bytes left in this filesystem. May be zero. */ @Override public long getAvailable() throws IOException { Thanks.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Yongjun Zhang Thanks a lot for review.. Updated the patch...Kindly review!!!

          Show
          brahmareddy Brahma Reddy Battula added a comment - Yongjun Zhang Thanks a lot for review.. Updated the patch...Kindly review!!!
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12723644/HDFS-5215-005.patch
          against trunk revision 75c5454.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestLeaseRecovery2

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10192//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10192//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12723644/HDFS-5215-005.patch against trunk revision 75c5454. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestLeaseRecovery2 Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10192//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10192//console This message is automatically generated.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Brahma Reddy Battula,

          Thanks for the new rev. I ran the failed test manually and it was sucessful.

          +1. Will commit later today if no further comments from other folks.

          Show
          yzhangal Yongjun Zhang added a comment - Hi Brahma Reddy Battula , Thanks for the new rev. I ran the failed test manually and it was sucessful. +1. Will commit later today if no further comments from other folks.
          Hide
          yzhangal Yongjun Zhang added a comment -

          I committed to trunk and branch-2.

          Thanks Brahma Reddy Battula for the contribution, and other folks for the review.

          Show
          yzhangal Yongjun Zhang added a comment - I committed to trunk and branch-2. Thanks Brahma Reddy Battula for the contribution, and other folks for the review.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7527 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7527/)
          HDFS-5215. dfs.datanode.du.reserved is not considered while computing (yzhang: rev 66763bb06f107f0e072c773a5feb25903c688ddc)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7527 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7527/ ) HDFS-5215 . dfs.datanode.du.reserved is not considered while computing (yzhang: rev 66763bb06f107f0e072c773a5feb25903c688ddc) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Yongjun Zhang Thanks a lot for review and commit...

          Show
          brahmareddy Brahma Reddy Battula added a comment - Yongjun Zhang Thanks a lot for review and commit...
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #157 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/157/)
          HDFS-5215. dfs.datanode.du.reserved is not considered while computing (yzhang: rev 66763bb06f107f0e072c773a5feb25903c688ddc)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #157 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/157/ ) HDFS-5215 . dfs.datanode.du.reserved is not considered while computing (yzhang: rev 66763bb06f107f0e072c773a5feb25903c688ddc) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2089 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2089/)
          HDFS-5215. dfs.datanode.du.reserved is not considered while computing (yzhang: rev 66763bb06f107f0e072c773a5feb25903c688ddc)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2089 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2089/ ) HDFS-5215 . dfs.datanode.du.reserved is not considered while computing (yzhang: rev 66763bb06f107f0e072c773a5feb25903c688ddc) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #148 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/148/)
          HDFS-5215. dfs.datanode.du.reserved is not considered while computing (yzhang: rev 66763bb06f107f0e072c773a5feb25903c688ddc)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #148 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/148/ ) HDFS-5215 . dfs.datanode.du.reserved is not considered while computing (yzhang: rev 66763bb06f107f0e072c773a5feb25903c688ddc) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #891 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/891/)
          HDFS-5215. dfs.datanode.du.reserved is not considered while computing (yzhang: rev 66763bb06f107f0e072c773a5feb25903c688ddc)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #891 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/891/ ) HDFS-5215 . dfs.datanode.du.reserved is not considered while computing (yzhang: rev 66763bb06f107f0e072c773a5feb25903c688ddc) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #158 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/158/)
          HDFS-5215. dfs.datanode.du.reserved is not considered while computing (yzhang: rev 66763bb06f107f0e072c773a5feb25903c688ddc)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #158 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/158/ ) HDFS-5215 . dfs.datanode.du.reserved is not considered while computing (yzhang: rev 66763bb06f107f0e072c773a5feb25903c688ddc) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2107 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2107/)
          HDFS-5215. dfs.datanode.du.reserved is not considered while computing (yzhang: rev 66763bb06f107f0e072c773a5feb25903c688ddc)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2107 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2107/ ) HDFS-5215 . dfs.datanode.du.reserved is not considered while computing (yzhang: rev 66763bb06f107f0e072c773a5feb25903c688ddc) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          Hide
          kihwal Kihwal Lee added a comment -

          I think it is worth fixing in 2.7.1. If no one objects, I will cherry pick.

          Show
          kihwal Kihwal Lee added a comment - I think it is worth fixing in 2.7.1. If no one objects, I will cherry pick.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Thanks Kihwal Lee, sure, please.

          Show
          yzhangal Yongjun Zhang added a comment - Thanks Kihwal Lee , sure, please.
          Hide
          kihwal Kihwal Lee added a comment -

          I've cherry-picked the change to branch-2.7.

          Show
          kihwal Kihwal Lee added a comment - I've cherry-picked the change to branch-2.7.
          Hide
          cnauroth Chris Nauroth added a comment -

          It appears this patch had a side effect of counting dfs.datanode.du.reserved space towards non-DFS used as reported by hdfs dfsadmin -report. Prior to this patch, reserved space did not count towards non-DFS used. Was this intentional, or was it an unintended side effect? My opinion is that the definition of non-DFS used should not have changed. Please let me know your thoughts, and if necessary, we can file a new follow-up jira.

          Show
          cnauroth Chris Nauroth added a comment - It appears this patch had a side effect of counting dfs.datanode.du.reserved space towards non-DFS used as reported by hdfs dfsadmin -report . Prior to this patch, reserved space did not count towards non-DFS used. Was this intentional, or was it an unintended side effect? My opinion is that the definition of non-DFS used should not have changed. Please let me know your thoughts, and if necessary, we can file a new follow-up jira.
          Hide
          cnauroth Chris Nauroth added a comment -

          I filed HDFS-9038 to propose restoring the prior definition of non-DFS used.

          Show
          cnauroth Chris Nauroth added a comment - I filed HDFS-9038 to propose restoring the prior definition of non-DFS used.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Sorry for coming late..Yes, it was not intentional..

          My opinion is that the definition of non-DFS used should not have changed.

          agree with you.

          Let's followup in HDFS-9038.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Sorry for coming late..Yes, it was not intentional.. My opinion is that the definition of non-DFS used should not have changed. agree with you. Let's followup in HDFS-9038 .

            People

            • Assignee:
              brahmareddy Brahma Reddy Battula
              Reporter:
              brahmareddy Brahma Reddy Battula
            • Votes:
              0 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development