Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2827

Cannot save namespace after renaming a directory above a file with an open lease

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.24.0
    • Fix Version/s: 0.24.0, 0.23.1, 1.2.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When i execute the following operations and wait for checkpoint to complete.

      fs.mkdirs(new Path("/test1"));
      FSDataOutputStream create = fs.create(new Path("/test/abc.txt")); //dont close
      fs.rename(new Path("/test/"), new Path("/test1/"));

      Check-pointing is failing with the following exception.

      2012-01-23 15:03:14,204 ERROR namenode.FSImage (FSImage.java:run(795)) - Unable to save image for E:\HDFS-1623\hadoop-hdfs-project\hadoop-hdfs\build\test\data\dfs\name3
      java.io.IOException: saveLeases found path /test1/est/abc.txt but no matching entry in namespace.[/test1/est/abc.txt]
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveFilesUnderConstruction(FSNamesystem.java:4336)
      at org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Saver.save(FSImageFormat.java:588)
      at org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:761)
      at org.apache.hadoop.hdfs.server.namenode.FSImage$FSImageSaver.run(FSImage.java:789)
      at java.lang.Thread.run(Unknown Source)

      1. HDFS-2827.patch
        4 kB
        Uma Maheswara Rao G
      2. hdfs-2827-b1.txt
        4 kB
        Eli Collins
      3. HDFS-2827-test.patch
        2 kB
        Aaron T. Myers

        Activity

        Hide
        Eli Collins added a comment -

        Forgot to mention, the findbugs warnings are HADOOP-7847.

        Show
        Eli Collins added a comment - Forgot to mention, the findbugs warnings are HADOOP-7847 .
        Eli Collins made changes -
        Fix Version/s 1.2.0 [ 12321657 ]
        Target Version/s 0.24.0 [ 12317653 ]
        Hide
        Eli Collins added a comment -

        Thanks for the review ATM, I've committed this to branch-1.

        Show
        Eli Collins added a comment - Thanks for the review ATM, I've committed this to branch-1.
        Hide
        Eli Collins added a comment -
         
             [exec] 
             [exec] -1 overall.  
             [exec] 
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec] 
             [exec]     +1 tests included.  The patch appears to include 6 new or modified tests.
             [exec] 
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec] 
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec] 
             [exec]     -1 findbugs.  The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings.
             [exec]
        

        findbugs are HADOOP-7847.

        Show
        Eli Collins added a comment - [exec] [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings. [exec] findbugs are HADOOP-7847 .
        Hide
        Aaron T. Myers added a comment -

        +1, the branch-1 patch looks good to me.

        Show
        Aaron T. Myers added a comment - +1, the branch-1 patch looks good to me.
        Eli Collins made changes -
        Attachment hdfs-2827-b1.txt [ 12535600 ]
        Hide
        Eli Collins added a comment -

        Patch for branch-1 attached. Is a straightforward backport of the trunk patch.

        Show
        Eli Collins added a comment - Patch for branch-1 attached. Is a straightforward backport of the trunk patch.
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Uma Maheswara Rao G added a comment -

        Hi Knostantin, this should be the bug in 0.22 as well.
        Please check, if its worth to put in that version as well. I can post the patch.

        Show
        Uma Maheswara Rao G added a comment - Hi Knostantin, this should be the bug in 0.22 as well. Please check, if its worth to put in that version as well. I can post the patch.
        Uma Maheswara Rao G made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Uma Maheswara Rao G made changes -
        Resolution Incomplete [ 4 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Hide
        Uma Maheswara Rao G added a comment -

        Sure i will take a look in some time.
        And Thanks alot for taking alook. So, i am marking it as fixed.

        Show
        Uma Maheswara Rao G added a comment - Sure i will take a look in some time. And Thanks alot for taking alook. So, i am marking it as fixed.
        Hide
        Aaron T. Myers added a comment -

        Hey Uma, I've elected not to revert the patch as-committed, since it does indeed fix the bug at hand. But, in reviewing the patch, I've come up with a number of ideas for how LeaseManager#changeLease and FSNamesystem#unprotectedChangeLease can be improved. I've filed them in this JIRA: HDFS-2875. If you would like to take up this work, that'd be great. Otherwise, hopefully someone else will improve this code when they've got some time.

        Show
        Aaron T. Myers added a comment - Hey Uma, I've elected not to revert the patch as-committed, since it does indeed fix the bug at hand. But, in reviewing the patch, I've come up with a number of ideas for how LeaseManager#changeLease and FSNamesystem#unprotectedChangeLease can be improved. I've filed them in this JIRA: HDFS-2875 . If you would like to take up this work, that'd be great. Otherwise, hopefully someone else will improve this code when they've got some time.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #976 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/976/)
        HDFS-2827. When the parent of a directory is the root, renaming the directory results in leases updated incorrectly. Contributed by Uma Maheswara Rao G

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238700
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #976 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/976/ ) HDFS-2827 . When the parent of a directory is the root, renaming the directory results in leases updated incorrectly. Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238700 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #178 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/178/)
        svn merge -c 1238700 from trunk for HDFS-2827.

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238709
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #178 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/178/ ) svn merge -c 1238700 from trunk for HDFS-2827 . szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238709 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #156 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/156/)
        svn merge -c 1238700 from trunk for HDFS-2827.

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238709
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #156 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/156/ ) svn merge -c 1238700 from trunk for HDFS-2827 . szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238709 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #943 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/943/)
        HDFS-2827. When the parent of a directory is the root, renaming the directory results in leases updated incorrectly. Contributed by Uma Maheswara Rao G

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238700
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #943 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/943/ ) HDFS-2827 . When the parent of a directory is the root, renaming the directory results in leases updated incorrectly. Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238700 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #467 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/467/)
        svn merge -c 1238700 from trunk for HDFS-2827.

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238709
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #467 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/467/ ) svn merge -c 1238700 from trunk for HDFS-2827 . szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238709 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1640 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1640/)
        HDFS-2827. When the parent of a directory is the root, renaming the directory results in leases updated incorrectly. Contributed by Uma Maheswara Rao G

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238700
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1640 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1640/ ) HDFS-2827 . When the parent of a directory is the root, renaming the directory results in leases updated incorrectly. Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238700 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Aaron, thanks for your understanding.

        Show
        Tsz Wo Nicholas Sze added a comment - Aaron, thanks for your understanding.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #443 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/443/)
        svn merge -c 1238700 from trunk for HDFS-2827.

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238709
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #443 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/443/ ) svn merge -c 1238700 from trunk for HDFS-2827 . szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238709 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        Hide
        Aaron T. Myers added a comment -

        Hi Aaron, sorry that I did not see you message earlier. You can still review it and, if you find any problem, please feel free to revert it.

        No biggie. I'll take a look now.

        BTW, the patch is available for more than a week and Uma has particularly asked you to take a look ...

        Yea, Uma also emailed me offline and I told him I would take a look at it, but it wasn't my highest priority at the moment because of some deadlines I had yesterday. Hence, I had intended to look today. I probably should've said that on this JIRA.

        Show
        Aaron T. Myers added a comment - Hi Aaron, sorry that I did not see you message earlier. You can still review it and, if you find any problem, please feel free to revert it. No biggie. I'll take a look now. BTW, the patch is available for more than a week and Uma has particularly asked you to take a look ... Yea, Uma also emailed me offline and I told him I would take a look at it, but it wasn't my highest priority at the moment because of some deadlines I had yesterday. Hence, I had intended to look today. I probably should've said that on this JIRA.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #452 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/452/)
        svn merge -c 1238700 from trunk for HDFS-2827.

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238709
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #452 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/452/ ) svn merge -c 1238700 from trunk for HDFS-2827 . szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238709 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        Tsz Wo Nicholas Sze made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.24.0 [ 12317653 ]
        Fix Version/s 0.23.1 [ 12318885 ]
        Resolution Incomplete [ 4 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have committed the patch to trunk and 0.23. Thanks, Uma!


        Hi Aaron, sorry that I did not see you message earlier. You can still review it and, if you find any problem, please feel free to revert it.

        BTW, the patch is available for more than a week and Uma has particularly asked you to take a look ...

        Show
        Tsz Wo Nicholas Sze added a comment - I have committed the patch to trunk and 0.23. Thanks, Uma! Hi Aaron, sorry that I did not see you message earlier. You can still review it and, if you find any problem, please feel free to revert it. BTW, the patch is available for more than a week and Uma has particularly asked you to take a look ...
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1623 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1623/)
        HDFS-2827. When the parent of a directory is the root, renaming the directory results in leases updated incorrectly. Contributed by Uma Maheswara Rao G

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238700
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1623 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1623/ ) HDFS-2827 . When the parent of a directory is the root, renaming the directory results in leases updated incorrectly. Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238700 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1695 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1695/)
        HDFS-2827. When the parent of a directory is the root, renaming the directory results in leases updated incorrectly. Contributed by Uma Maheswara Rao G

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238700
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1695 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1695/ ) HDFS-2827 . When the parent of a directory is the root, renaming the directory results in leases updated incorrectly. Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238700 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        Hide
        Aaron T. Myers added a comment -

        Nicholas, can you give me a day to review this before commit? I'd like to take a look but don't have the time right this moment.

        Show
        Aaron T. Myers added a comment - Nicholas, can you give me a day to review this before commit? I'd like to take a look but don't have the time right this moment.
        Tsz Wo Nicholas Sze made changes -
        Hadoop Flags Reviewed [ 10343 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 good catch.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 good catch.
        Hide
        Uma Maheswara Rao G added a comment -

        Aaron, could you please take a look?

        Show
        Uma Maheswara Rao G added a comment - Aaron, could you please take a look?
        Hide
        Uma Maheswara Rao G added a comment -

        findbugs and javadoc warnings are unrelated to this patch!

        Show
        Uma Maheswara Rao G added a comment - findbugs and javadoc warnings are unrelated to this patch!
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12511688/HDFS-2827.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        -1 javadoc. The javadoc tool appears to have generated 21 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1803//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1803//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1803//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12511688/HDFS-2827.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 21 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1803//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1803//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1803//console This message is automatically generated.
        Uma Maheswara Rao G made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Uma Maheswara Rao G added a comment -

        Here is a simple patch to handle this issue.
        One extra path separator will be appended when source path parent is root directory. When forming the destination directory, substring is creating the problem due to this extra path separator.

        Show
        Uma Maheswara Rao G added a comment - Here is a simple patch to handle this issue. One extra path separator will be appended when source path parent is root directory. When forming the destination directory, substring is creating the problem due to this extra path separator.
        Uma Maheswara Rao G made changes -
        Attachment HDFS-2827.patch [ 12511688 ]
        Hide
        Uma Maheswara Rao G added a comment -

        Thanks a lot Aaron for digging into it.

        but in doing so I also confirmed that this is not specific to the HA branch or the Standby NN. As such, I'm going to convert this to a top-level issue.

        Agree. I just filed it in HA branch because i was doing some HA flows test. This is causing checkpointing failure.

        Will take a look into it.

        Thanks
        Uma

        Show
        Uma Maheswara Rao G added a comment - Thanks a lot Aaron for digging into it. but in doing so I also confirmed that this is not specific to the HA branch or the Standby NN. As such, I'm going to convert this to a top-level issue. Agree. I just filed it in HA branch because i was doing some HA flows test. This is causing checkpointing failure. Will take a look into it. Thanks Uma
        Aaron T. Myers made changes -
        Summary Checkpointing is failing with saveleases found but no entry in namespace. Cannot save namespace after renaming a directory above a file with an open lease
        Aaron T. Myers made changes -
        Affects Version/s 0.24.0 [ 12317653 ]
        Affects Version/s HA branch (HDFS-1623) [ 12317568 ]
        Target Version/s 0.24.0 [ 12317653 ]
        Environment Dev env - Windows
        Component/s name-node [ 12312926 ]
        Component/s ha [ 12316609 ]
        Aaron T. Myers made changes -
        Attachment HDFS-2827-test.patch [ 12511638 ]
        Hide
        Aaron T. Myers added a comment -

        Test which demonstrates the issue.

        Show
        Aaron T. Myers added a comment - Test which demonstrates the issue.
        Aaron T. Myers made changes -
        Parent HDFS-1623 [ 12498318 ]
        Issue Type Sub-task [ 7 ] Bug [ 1 ]
        Hide
        Aaron T. Myers added a comment -

        Hey Uma, I confirmed that this issue does exist on Linux as well, but in doing so I also confirmed that this is not specific to the HA branch or the Standby NN. As such, I'm going to convert this to a top-level issue.

        I'll attach a patch in a moment with a test case that demonstrates the issue.

        Show
        Aaron T. Myers added a comment - Hey Uma, I confirmed that this issue does exist on Linux as well, but in doing so I also confirmed that this is not specific to the HA branch or the Standby NN. As such, I'm going to convert this to a top-level issue. I'll attach a patch in a moment with a test case that demonstrates the issue.
        Uma Maheswara Rao G made changes -
        Parent HDFS-1623 [ 12498318 ]
        Issue Type Bug [ 1 ] Sub-task [ 7 ]
        Hide
        Uma Maheswara Rao G added a comment -

        I did not debug the issue yet. When i am testing some HA flows, met this issue.
        This is looks like some path construction issue in windows.
        Currently marked it as Major. Once i confirm that this issue is fine in Linux, then i will make it as minor.

        Show
        Uma Maheswara Rao G added a comment - I did not debug the issue yet. When i am testing some HA flows, met this issue. This is looks like some path construction issue in windows. Currently marked it as Major. Once i confirm that this issue is fine in Linux, then i will make it as minor.
        Uma Maheswara Rao G made changes -
        Field Original Value New Value
        Summary Next checkpointing is failing with saveleases found but no entry in namespace. Checkpointing is failing with saveleases found but no entry in namespace.
        Uma Maheswara Rao G created issue -

          People

          • Assignee:
            Uma Maheswara Rao G
            Reporter:
            Uma Maheswara Rao G
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development