Hadoop Common
  1. Hadoop Common
  2. HADOOP-6963

Fix FileUtil.getDU. It should not include the size of the directory or follow symbolic links

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.20.205.0, 0.23.1
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      The getDU method should not include the size of the directory. The Java interface says that the value is undefined and in Linux/Sun it gets the 4096 for the inode. Clearly this isn't useful.
      It also recursively calls itself. In case the directory has a symbolic link forming a cycle, getDU keeps spinning in the cycle. In our case, we saw this in the org.apache.hadoop.mapred.JobLocalizer.downloadPrivateCacheObjects call. This prevented other tasks on the same node from committing, causing the TT to become effectively useless (because the JT thinks it already has enough tasks running)

      1. HADOOP-6963.branch-23.patch
        3 kB
        Ravi Prakash
      2. HADOOP-6963.branch-1.0.2.patch
        3 kB
        Ravi Prakash
      3. HADOOP-6963.branch-23.patch
        3 kB
        Robert Joseph Evans
      4. HADOOP-6963.branch-23.patch
        3 kB
        Ravi Prakash
      5. HADOOP-6963.branch-1.0.2.patch
        3 kB
        Ravi Prakash
      6. HADOOP-6963.branch-1.patch
        3 kB
        Ravi Prakash

        Activity

        Hide
        Ravi Prakash added a comment -

        @Owen : I took the liberty of editing this JIRA to include another fix in the same method.

        Show
        Ravi Prakash added a comment - @Owen : I took the liberty of editing this JIRA to include another fix in the same method.
        Hide
        Ravi Prakash added a comment -

        This patch applies to branch-0.23. Will soon upload one for 0.20 and trunk (if its different)

        Show
        Ravi Prakash added a comment - This patch applies to branch-0.23. Will soon upload one for 0.20 and trunk (if its different)
        Hide
        Ravi Prakash added a comment -

        Uploading a patch that applies on top of branch-1.0.2. Can someone please review and commit this?

        The patch for branch-0.23 also applies on trunk.

        Show
        Ravi Prakash added a comment - Uploading a patch that applies on top of branch-1.0.2. Can someone please review and commit this? The patch for branch-0.23 also applies on trunk.
        Hide
        Ravi Prakash added a comment -

        Can some kind committer please review and commit the patches?

        Show
        Ravi Prakash added a comment - Can some kind committer please review and commit the patches?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12520989/HADOOP-6963.branch-1.0.2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/811//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520989/HADOOP-6963.branch-1.0.2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/811//console This message is automatically generated.
        Hide
        Ravi Prakash added a comment -

        Hadoop QA could not apply the latest patch (meant for branch-1.0.2) to trunk.

        The patches apply perfectly fine to their targets. Can committers please review and commit these?

        Show
        Ravi Prakash added a comment - Hadoop QA could not apply the latest patch (meant for branch-1.0.2) to trunk. The patches apply perfectly fine to their targets. Can committers please review and commit these?
        Hide
        Ravi Prakash added a comment -

        Hadoop QA could not apply the latest patch (meant for branch-1.0.2) to trunk.

        The patches apply perfectly fine to their targets. Can committers please review and commit these?

        Show
        Ravi Prakash added a comment - Hadoop QA could not apply the latest patch (meant for branch-1.0.2) to trunk. The patches apply perfectly fine to their targets. Can committers please review and commit these?
        Hide
        Robert Joseph Evans added a comment -

        Kicking Jenkins

        Show
        Robert Joseph Evans added a comment - Kicking Jenkins
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12521370/HADOOP-6963.branch-23.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/819//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/819//artifact/trunk/hadoop-common-project/patchprocess/newPatchFindbugsWarningshadoop-common.html
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/819//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521370/HADOOP-6963.branch-23.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/819//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/819//artifact/trunk/hadoop-common-project/patchprocess/newPatchFindbugsWarningshadoop-common.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/819//console This message is automatically generated.
        Hide
        Ravi Prakash added a comment -

        Thanks Bobby! The findbugs warning is because I am ignoring an IOException. I can do that because I am already setting isSymLink to false during initialization.

        Show
        Ravi Prakash added a comment - Thanks Bobby! The findbugs warning is because I am ignoring an IOException. I can do that because I am already setting isSymLink to false during initialization.
        Hide
        Robert Joseph Evans added a comment -

        I can see that, but we need to filter it either by bumping the findbugs count in hadoop-common-project/dev-support/test-patch.properties

        or by adding something to

        hadoop-common-project/hadoop-common/pom.xml

        similar to

            </plugin>
              <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>findbugs-maven-plugin</artifactId>
                 <configuration>
                  <findbugsXmlOutput>true</findbugsXmlOutput>
                  <xmlOutput>true</xmlOutput>
                  <excludeFilterFile>${mr.examples.basedir}/dev-support/findbugs-exclude.xml</excludeFilterFile>
                  <effort>Max</effort>
                </configuration>
              </plugin>
        

        I would prefer the second because it is less likely to mask other errors, but I am fine either way.

        Please also update the 1.0.2 patch similarly so test-patch will work there too.

        Show
        Robert Joseph Evans added a comment - I can see that, but we need to filter it either by bumping the findbugs count in hadoop-common-project/dev-support/test-patch.properties or by adding something to hadoop-common-project/hadoop-common/pom.xml similar to </plugin> <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>findbugs-maven-plugin</artifactId> <configuration> <findbugsXmlOutput> true </findbugsXmlOutput> <xmlOutput> true </xmlOutput> <excludeFilterFile>${mr.examples.basedir}/dev-support/findbugs-exclude.xml</excludeFilterFile> <effort>Max</effort> </configuration> </plugin> I would prefer the second because it is less likely to mask other errors, but I am fine either way. Please also update the 1.0.2 patch similarly so test-patch will work there too.
        Hide
        Ravi Prakash added a comment -

        Thanks Bobby! I modofied my own code to include a statement in the catch clause. I ran findbugs on my local computer and it did not generate any new warnings. Hopefully this should pass

        Show
        Ravi Prakash added a comment - Thanks Bobby! I modofied my own code to include a statement in the catch clause. I ran findbugs on my local computer and it did not generate any new warnings. Hopefully this should pass
        Hide
        Ravi Prakash added a comment -

        Uploading a new patch for 1.0.2 with the same change to get rid of the findbugs warning

        Show
        Ravi Prakash added a comment - Uploading a new patch for 1.0.2 with the same change to get rid of the findbugs warning
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12521513/HADOOP-6963.branch-23.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/826//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/826//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521513/HADOOP-6963.branch-23.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/826//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/826//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12521518/HADOOP-6963.branch-1.0.2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/827//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521518/HADOOP-6963.branch-1.0.2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/827//console This message is automatically generated.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1

        Show
        Tsz Wo Nicholas Sze added a comment - +1
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2080 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2080/)
        HADOOP-6963. In FileUtil.getDU(..), neither include the size of directories nor follow symbolic links. Contributed by Ravi Prakash (Revision 1309994)

        Result = SUCCESS
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309994
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2080 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2080/ ) HADOOP-6963 . In FileUtil.getDU(..), neither include the size of directories nor follow symbolic links. Contributed by Ravi Prakash (Revision 1309994) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309994 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have committed this to trunk and branch-2. Thanks, Ravi!

        (The branch-1 patch did not apply and so I skipped it. Please post a new patch if necessary.)

        Show
        Tsz Wo Nicholas Sze added a comment - I have committed this to trunk and branch-2. Thanks, Ravi! (The branch-1 patch did not apply and so I skipped it. Please post a new patch if necessary.)
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2005 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2005/)
        HADOOP-6963. In FileUtil.getDU(..), neither include the size of directories nor follow symbolic links. Contributed by Ravi Prakash (Revision 1309994)

        Result = SUCCESS
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309994
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2005 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2005/ ) HADOOP-6963 . In FileUtil.getDU(..), neither include the size of directories nor follow symbolic links. Contributed by Ravi Prakash (Revision 1309994) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309994 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2017 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2017/)
        HADOOP-6963. In FileUtil.getDU(..), neither include the size of directories nor follow symbolic links. Contributed by Ravi Prakash (Revision 1309994)

        Result = SUCCESS
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309994
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2017 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2017/ ) HADOOP-6963 . In FileUtil.getDU(..), neither include the size of directories nor follow symbolic links. Contributed by Ravi Prakash (Revision 1309994) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309994 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
        Hide
        Robert Joseph Evans added a comment -

        I pulled this into 0.23.3 too.

        Show
        Robert Joseph Evans added a comment - I pulled this into 0.23.3 too.
        Hide
        Ravi Prakash added a comment -

        Thanks Nicholas and Bobby! Bobby pointed out to me that the patch must apply to branch-1 . I'd originally targetted branch-1.0.2. Silly me.

        Could you please commit this updated patch too Nicholas?

        Show
        Ravi Prakash added a comment - Thanks Nicholas and Bobby! Bobby pointed out to me that the patch must apply to branch-1 . I'd originally targetted branch-1.0.2. Silly me. Could you please commit this updated patch too Nicholas?
        Hide
        Ravi Prakash added a comment -

        Reopening for committing to branch-1.

        Show
        Ravi Prakash added a comment - Reopening for committing to branch-1.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2018 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2018/)
        Moved HADOOP-6963 to 0.23.3 (Revision 1310022)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310022
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2018 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2018/ ) Moved HADOOP-6963 to 0.23.3 (Revision 1310022) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310022 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        Tsz Wo Nicholas Sze added a comment -

        > Could you please commit this updated patch too Nicholas?

        Sure. Thanks for posting a patch for branch-1. +1 it looks good.

        Show
        Tsz Wo Nicholas Sze added a comment - > Could you please commit this updated patch too Nicholas? Sure. Thanks for posting a patch for branch-1. +1 it looks good.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I also committed the patch to branch-1 and branch-1.0.

        Show
        Tsz Wo Nicholas Sze added a comment - I also committed the patch to branch-1 and branch-1.0.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2006 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2006/)
        Moved HADOOP-6963 to 0.23.3 (Revision 1310022)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310022
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2006 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2006/ ) Moved HADOOP-6963 to 0.23.3 (Revision 1310022) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310022 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2081 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2081/)
        Moved HADOOP-6963 to 0.23.3 (Revision 1310022)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310022
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2081 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2081/ ) Moved HADOOP-6963 to 0.23.3 (Revision 1310022) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310022 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #219 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/219/)
        HADOOP-6963. In FileUtil.getDU(..), neither include the size of directories nor follow symbolic links. (Ravi Prakash via bobby) (Revision 1310018)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310018
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/pom.xml
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #219 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/219/ ) HADOOP-6963 . In FileUtil.getDU(..), neither include the size of directories nor follow symbolic links. (Ravi Prakash via bobby) (Revision 1310018) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310018 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1006 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1006/)
        Moved HADOOP-6963 to 0.23.3 (Revision 1310022)
        HADOOP-6963. In FileUtil.getDU(..), neither include the size of directories nor follow symbolic links. Contributed by Ravi Prakash (Revision 1309994)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310022
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309994
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1006 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1006/ ) Moved HADOOP-6963 to 0.23.3 (Revision 1310022) HADOOP-6963 . In FileUtil.getDU(..), neither include the size of directories nor follow symbolic links. Contributed by Ravi Prakash (Revision 1309994) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310022 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309994 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1041 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1041/)
        Moved HADOOP-6963 to 0.23.3 (Revision 1310022)
        HADOOP-6963. In FileUtil.getDU(..), neither include the size of directories nor follow symbolic links. Contributed by Ravi Prakash (Revision 1309994)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310022
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309994
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1041 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1041/ ) Moved HADOOP-6963 to 0.23.3 (Revision 1310022) HADOOP-6963 . In FileUtil.getDU(..), neither include the size of directories nor follow symbolic links. Contributed by Ravi Prakash (Revision 1309994) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1310022 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309994 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
        Hide
        Ravi Prakash added a comment -

        I just wanted to come back and say this was a very peculiar case too.

        • The cycle has to be nested within another cycle. Inside the inner cycle an ELOOP is thrown by the kernel, which (I don't understand how) causes it to abandon that branch and come back to the outer loop. This probably also clears the stack, so that the JVM stack doesn't exhaust the memory either. (In our case, the process had been running for 9 days without getting an OOME).
        Show
        Ravi Prakash added a comment - I just wanted to come back and say this was a very peculiar case too. The cycle has to be nested within another cycle. Inside the inner cycle an ELOOP is thrown by the kernel, which (I don't understand how) causes it to abandon that branch and come back to the outer loop. This probably also clears the stack, so that the JVM stack doesn't exhaust the memory either. (In our case, the process had been running for 9 days without getting an OOME).
        Hide
        Ravi Prakash added a comment -

        A guess at why an Exception is not thrown when an ELOOP is received. File.exists() is the first method called with the path which should trigger the ELOOP and this is probably returning false which then causes the recursion to unroll until the outer loop.

        Show
        Ravi Prakash added a comment - A guess at why an Exception is not thrown when an ELOOP is received. File.exists() is the first method called with the path which should trigger the ELOOP and this is probably returning false which then causes the recursion to unroll until the outer loop.
        Hide
        Matt Foley added a comment -

        Closed upon release of Hadoop-1.0.3.

        Show
        Matt Foley added a comment - Closed upon release of Hadoop-1.0.3.
        Hide
        Luke Lu added a comment -

        Looks like we forgot to update the upgraded commons-io version in hadoop-core-pom-template.xml, which breaks downstream projects (e.g. HBase).

        Show
        Luke Lu added a comment - Looks like we forgot to update the upgraded commons-io version in hadoop-core-pom-template.xml, which breaks downstream projects (e.g. HBase).
        Hide
        Ravi Prakash added a comment -

        I see HADOOP-8417 has been filed! Thanks for reporting this Luke

        Show
        Ravi Prakash added a comment - I see HADOOP-8417 has been filed! Thanks for reporting this Luke

          People

          • Assignee:
            Ravi Prakash
            Reporter:
            Owen O'Malley
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development