Fix can be

long available = usage.getAvailable() - reserved;

Please let me know any comments on this..

I think its a nice finding Brahma, planning to post a patch?

You can also add a test which can detect this issue. We may not find this issue until you have some unuseable space in the partition right?

Uma Mahesh Thanks for taking a look at this issue..

Will post the patch with testcase.

Tested manually, after defect fix..it's working fine

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12610538/HDFS-5215.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5291//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5291//console

This message is automatically generated.

Here is the branch-2 code:

  /**
   * Calculate the capacity of the filesystem, after removing any
   * reserved capacity.
   * @return the unreserved number of bytes left in this filesystem. May be zero.
   */
  long getCapacity() {
    long remaining = usage.getCapacity() - reserved;
    return remaining > 0 ? remaining : 0;
  }

  @Override
  public long getAvailable() throws IOException {
    long remaining = getCapacity()-getDfsUsed();
    long available = usage.getAvailable();
    if (remaining > available) {
      remaining = available;
    }
    return (remaining > 0) ? remaining : 0;
  }

The logic seems to be correct.

• remaining is total capacity - reserved - dfs used. That is the space available for DFS use without considering actual space available on the disk.
• available is space currently available.

The remaining space returned is smaller of the above two. This seems correct to me. What is the need to remove reserved space from available above?

The remaining space returned is smaller of the above two. This seems correct to me. What is the need to remove reserved space from available above?

AS we are assigning the available to remaining,there is one case where always remaining will be greater than available space.. this scenario will come where dfsused is very less but disk got filled with non-dfs data..

  if (remaining > available) {
      remaining = available;
    }

Check following for same..

TEST ENV: total space of datanode disk is 69G, revered space 10G, available space is 66G at beginning, dd command generate files use 52G space.
Debug the old code：
long remaining = getCapacity()-getDfsUsed(); – 63242600448
long available = usage.getAvailable(); – 14144172032
if (remaining > available)

Upload 6 files, as per file has 666266906 bytes：
long remaining = getCapacity()-getDfsUsed(); – 59213767320
long available = usage.getAvailable(); – 10110709760
if (remaining > available)

Upload 7th file success.
Clear the env and degug the new code：
long remaining = getCapacity()-getDfsUsed(); – 63238742828
long available = usage.getAvailable()- reserved; – 3406753792
if (remaining > available)

Upload 6 files, as per file has 666266906 bytes：
long remaining = getCapacity()-getDfsUsed(); – 59881381888
long available = usage.getAvailable()- reserved; – 45535232
if (remaining > available)

Upload 7th file failed.

Have you considered that the underlying filesystem reserved space (ext3, ext4 -> tune2fs) can also affect this behavior?
I'm not sure if this is a problem in hadoop-3.0.x anymore.

I recently opened a jira regarding this issue: https://issues.apache.org/jira/browse/HDFS-5926

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12610538/HDFS-5215.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. There were no new javadoc warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6587//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6587//console

This message is automatically generated.

Hi Alexander Fahlke,
Thanks for taking look into this issue..

Have you considered that the underlying filesystem reserved space (ext3, ext4 -> tune2fs) can also affect this behavior?

Yes, I checked in ext3 file system only..

I'm not sure if this is a problem in hadoop-3.0.x anymore.

As I knew , it is not fixed..problem will be present in hadoop-3.0.x . .Please let me know your thoughts on this...

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12610538/HDFS-5215.patch
against trunk revision 14e2639.

-1 patch. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8019//console

This message is automatically generated.

Any thoughts on this jira..?

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12610538/HDFS-5215.patch
against trunk revision ed70fa1.

-1 patch. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9736//console

This message is automatically generated.

HI Brahma Reddy Battula, thanks for reporting the issue and the patch here. The patch doesn't apply anymore. would you upload a new rev? Thanks.

HI Suresh Srinivas,

Brahma's reply to your comment

https://issues.apache.org/jira/browse/HDFS-5215?focusedCommentId=13820096&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13820096

seems reasonable to me. Would you please comment?

Thanks.

Hm, seems it's better we deduct reserved from DF.java:

 public long getAvailable() {
    return dirFile.getUsableSpace();
  }

That is, replace the above with {{ dirFile.getUsableSpace() - reserved; }}, if we define DF#getAvailable() as the space available to HDFS.

Thanks.

DF#getAvailable() is called by LocalDirAllocator#getLocalPathForWrite and others, so I think the change I suggested in my last comment would make the patch more comprehensive. Thanks.

Thanks a lot for taking a look into this issue..Yes, I am waiting for Suresh Srinivas reply..

Sure Brahma. The patch doesn't apply anymore. I also have some review comments down there, wonder if you could submit an updated patch, so people have a new base to discuss? thanks.

HI Brahma Reddy Battula,

I noticed that I did not ping your name when I did the following comment, I'm putting it here again, thanks for following up.

The patch doesn't apply anymore. I also have some review comments, wonder if you could submit an updated patch, so people have a new base to discuss? thanks

Hi Brahma Reddy Battula,

I studied the callers of DF#getAvailable() including LocalDirAllocator#getLocalPathForWrite a little more and realize that they don't need to be aware of the reserved space specified by dfs.datanode.du.reserved. So we don't need to touch the implementation in DF.java.

Sorry for the forth and back.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12707741/HDFS-5215-002.patch
against trunk revision af618f2.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. There were no new javadoc warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The test build failed in hadoop-hdfs-project/hadoop-hdfs

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10088//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10088//console

This message is automatically generated.

Testcase failures are unrelated this patch..@Yongjun Zhang Kindly Review..

HI Brahma Reddy Battula,

Thanks for updating the patch. It looks good to me, some comments:

1. add javadoc for getAvailable(), like how getCapacity() method does.
2. add comment to, the freeSpace is now excluding reserved after this patch:

   private static class VolumeInfo {
       final String directory;
       final long usedSpace;
       final long freeSpace;
       final long reservedSpace;
   

3. if possible, can we add a test or modify TestNamenodeCapacityReport#testVolumeSize to see the impact of this change?

Thanks.

Thanks a lot for review...Updated the patch based on your comments..Tested manually which I mentioned earlier in this jira.getraimining() will be called in TestNamenodeCapacityReport#testVolumeSize which will internally calls getavailable(), Hence I did not updated this.. Hopefully this should be ok..

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12708742/HDFS-5215-003.patch
against trunk revision 4922394.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

-1 javac. The applied patch generated 1153 javac compiler warnings (more than the trunk's current 1151 warnings).

+1 javadoc. There were no new javadoc warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs:

org.apache.hadoop.mapreduce.v2.app.TestMRClientService
org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryParsing

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10143//testReport/
Javac warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/10143//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10143//console

This message is automatically generated.

Testcase failures and javadoc wars are unrelated this patch.

Shouldn't we also subtract rbwReserved ?

Thanks for Review,, I will update patch..

Updated the patch...Yongjun Zhang kindly review...Thanks..

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12709278/HDFS-5215-004.patch
against trunk revision db80e42.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. There were no new javadoc warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics

The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10174//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10174//console

This message is automatically generated.

Hi Brahma Reddy Battula,

Sorry for the delay, I was buried in other stuff.

I agree with zhouyingchao that including rbwReserved is reasonable. Suggest to change the comment

1.
remove {{* the freeSpace is now excluding reserved + rbw after HDFS-5215 }} and add comments like blow

   final long usedSpace;     // size of space used by HDFS
   final long freeSpace;     // size of free space excluding reserved space
   final long reservedSpace; // size of space reserved for non-HDFS and RBW

2.

 /**
   * Return either the configured capacity of the file system if configured;
   * or the capacity of the file system excluding space reserved for non-HDFS.
   * @return the unreserved number of bytes left in this filesystem. May be zero.
   */
  @VisibleForTesting
  public long getCapacity() {

3.

 /*
   * Calculate the available space of the filesystem, excluding space reserved
   * for non-HDFS and space reserved for RBW
   * 
   * @return the available number of bytes left in this filesystem. May be zero.
   */
  @Override
  public long getAvailable() throws IOException {

Thanks.

Yongjun Zhang Thanks a lot for review.. Updated the patch...Kindly review!!!

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12723644/HDFS-5215-005.patch
against trunk revision 75c5454.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. There were no new javadoc warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

org.apache.hadoop.hdfs.TestLeaseRecovery2

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10192//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10192//console

This message is automatically generated.

Hi Brahma Reddy Battula,

Thanks for the new rev. I ran the failed test manually and it was sucessful.

+1. Will commit later today if no further comments from other folks.

I committed to trunk and branch-2.

Thanks Brahma Reddy Battula for the contribution, and other folks for the review.

FAILURE: Integrated in Hadoop-trunk-Commit #7527 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7527/)
HDFS-5215. dfs.datanode.du.reserved is not considered while computing (yzhang: rev 66763bb06f107f0e072c773a5feb25903c688ddc)

• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java

Yongjun Zhang Thanks a lot for review and commit...

FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #157 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/157/)
HDFS-5215. dfs.datanode.du.reserved is not considered while computing (yzhang: rev 66763bb06f107f0e072c773a5feb25903c688ddc)

• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java

FAILURE: Integrated in Hadoop-Hdfs-trunk #2089 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2089/)
HDFS-5215. dfs.datanode.du.reserved is not considered while computing (yzhang: rev 66763bb06f107f0e072c773a5feb25903c688ddc)

• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java

FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #148 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/148/)
HDFS-5215. dfs.datanode.du.reserved is not considered while computing (yzhang: rev 66763bb06f107f0e072c773a5feb25903c688ddc)

• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

FAILURE: Integrated in Hadoop-Yarn-trunk #891 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/891/)
HDFS-5215. dfs.datanode.du.reserved is not considered while computing (yzhang: rev 66763bb06f107f0e072c773a5feb25903c688ddc)

• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #158 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/158/)
HDFS-5215. dfs.datanode.du.reserved is not considered while computing (yzhang: rev 66763bb06f107f0e072c773a5feb25903c688ddc)

• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

FAILURE: Integrated in Hadoop-Mapreduce-trunk #2107 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2107/)
HDFS-5215. dfs.datanode.du.reserved is not considered while computing (yzhang: rev 66763bb06f107f0e072c773a5feb25903c688ddc)

• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java

I think it is worth fixing in 2.7.1. If no one objects, I will cherry pick.

Thanks Kihwal Lee, sure, please.

I've cherry-picked the change to branch-2.7.

It appears this patch had a side effect of counting dfs.datanode.du.reserved space towards non-DFS used as reported by hdfs dfsadmin -report. Prior to this patch, reserved space did not count towards non-DFS used. Was this intentional, or was it an unintended side effect? My opinion is that the definition of non-DFS used should not have changed. Please let me know your thoughts, and if necessary, we can file a new follow-up jira.

I filed HDFS-9038 to propose restoring the prior definition of non-DFS used.

Sorry for coming late..Yes, it was not intentional..

My opinion is that the definition of non-DFS used should not have changed.

agree with you.

Let's followup in HDFS-9038.