Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1881

Federation: after taking snapshot the current directory of datanode is empty

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      After taking a snapshot in Federation (by starting up namenode with option -upgrade), it appears that the current directory of data node does not contain the block files. We have also verified that upgrading from 20 to Federation does not have this problem.

      1. HDFS-1881.2.patch
        5 kB
        Tanping Wang
      2. HDFS-1881.patch
        5 kB
        Tanping Wang

        Activity

        Hide
        Tanping Wang added a comment -

        The problem was that when doing upgrade for BlockPoolSliceStorage we were hard linking between incorrect directories.

        Show
        Tanping Wang added a comment - The problem was that when doing upgrade for BlockPoolSliceStorage we were hard linking between incorrect directories.
        Hide
        Suresh Srinivas added a comment -

        Patch looks good. Thanks for catching the bug. One minor comment:
        In tests use MiniDFSCluster#getFinalizedDir() instead of building it using various path fragments. There are example in other tests.

        Show
        Suresh Srinivas added a comment - Patch looks good. Thanks for catching the bug. One minor comment: In tests use MiniDFSCluster#getFinalizedDir() instead of building it using various path fragments. There are example in other tests.
        Hide
        Tanping Wang added a comment -

        Upload a patch to address the review comments.

        Show
        Tanping Wang added a comment - Upload a patch to address the review comments.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12479183/HDFS-1881.2.patch
        against trunk revision 1102948.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.cli.TestHDFSCLI
        org.apache.hadoop.hdfs.server.balancer.TestBalancer
        org.apache.hadoop.hdfs.TestDFSShell
        org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
        org.apache.hadoop.hdfs.TestFileConcurrentReader
        org.apache.hadoop.tools.TestJMXGet

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/526//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/526//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/526//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479183/HDFS-1881.2.patch against trunk revision 1102948. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.server.balancer.TestBalancer org.apache.hadoop.hdfs.TestDFSShell org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.tools.TestJMXGet +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/526//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/526//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/526//console This message is automatically generated.
        Hide
        Tanping Wang added a comment -

        Have seen the unit test failure on bulid#510,#511, #525 etc. And they are unrelated to this patch.

        Show
        Tanping Wang added a comment - Have seen the unit test failure on bulid#510,#511, #525 etc. And they are unrelated to this patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12479183/HDFS-1881.2.patch
        against trunk revision 1103987.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
        org.apache.hadoop.hdfs.TestFileConcurrentReader
        org.apache.hadoop.tools.TestJMXGet

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/535//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/535//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/535//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479183/HDFS-1881.2.patch against trunk revision 1103987. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.tools.TestJMXGet +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/535//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/535//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/535//console This message is automatically generated.
        Hide
        Suresh Srinivas added a comment -

        I committed the patch.

        Show
        Suresh Srinivas added a comment - I committed the patch.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #660 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/660/)
        HDFS-1881. Federation: after taking snapshot the current directory of datanode is empty. Contributed by Tanping Wang.

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #660 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/660/ ) HDFS-1881 . Federation: after taking snapshot the current directory of datanode is empty. Contributed by Tanping Wang.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #673 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/673/)

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #673 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/673/ )

          People

          • Assignee:
            Tanping Wang
            Reporter:
            Tanping Wang
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development