Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2223

Untangle depencencies between NN components

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 2.0.0-alpha
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog

      1. hdfs-2223-10.txt
        54 kB
        Todd Lipcon
      2. hdfs-2223-9.txt
        54 kB
        Todd Lipcon
      3. hdfs-2223-8.txt
        53 kB
        Todd Lipcon
      4. hdfs-2223-7.txt
        53 kB
        Todd Lipcon
      5. hdfs-2223-6.txt
        51 kB
        Suresh Srinivas
      6. hdfs-2223-5.txt
        50 kB
        Todd Lipcon
      7. hdfs-2223-4.txt
        46 kB
        Todd Lipcon
      8. hdfs-2223-3.txt
        46 kB
        Aaron T. Myers
      9. hdfs-2223-2.txt
        46 kB
        Todd Lipcon
      10. hdfs-2223-1.txt
        44 kB
        Todd Lipcon

        Activity

        Hide
        Todd Lipcon added a comment -

        First, here's my conception of what the responsibilities and interactions are meant to be:

        NameNode: container class that handles startup/shutdown of the Java process itself. Starts/stops RPC and HTTP interfaces.
        FSNamesystem: container of both transient and persisted namespace state:
        – container for BlockManager, DatanodeManager, DelegationTokens, LeaseManager, etc
        – RPC calls that modify or inspect the namespace should get delegated here
        — anything that touches only blocks (eg block reports) should get delegated to BlockManager
        — anything that touches only file information (eg permissions, mkdirs) should get delegated to FSDirectory
        — anything that crosses two of the above components should be coordinated here
        – should log mutations to FSEditLog
        FSDirectory:
        – Namespace state that persists across restarts: the tree of inodes, and for each inode, the block list
        – should log mutations to FSEditLog
        FSImage: responsible for the mechanisms by which an FSNamesystem is snapshotted and logged
        – should be responsible for lifecycle of FSEditLog and NNStorage
        NNStorage: should only be responsible for file management inside storage directories, and tracking failures states of those directories
        – should not have any references "up" to other classes
        FSEditLog: responsible for managing a set of journals
        – after initialization, should not need to reference any other classes

        Hopefully this jives with what other people understand to be the "ideal" we're going for.

        Show
        Todd Lipcon added a comment - First, here's my conception of what the responsibilities and interactions are meant to be: NameNode : container class that handles startup/shutdown of the Java process itself. Starts/stops RPC and HTTP interfaces. FSNamesystem : container of both transient and persisted namespace state: – container for BlockManager, DatanodeManager, DelegationTokens, LeaseManager, etc – RPC calls that modify or inspect the namespace should get delegated here — anything that touches only blocks (eg block reports) should get delegated to BlockManager — anything that touches only file information (eg permissions, mkdirs) should get delegated to FSDirectory — anything that crosses two of the above components should be coordinated here – should log mutations to FSEditLog FSDirectory : – Namespace state that persists across restarts: the tree of inodes, and for each inode, the block list – should log mutations to FSEditLog FSImage : responsible for the mechanisms by which an FSNamesystem is snapshotted and logged – should be responsible for lifecycle of FSEditLog and NNStorage NNStorage : should only be responsible for file management inside storage directories, and tracking failures states of those directories – should not have any references "up" to other classes FSEditLog : responsible for managing a set of journals – after initialization, should not need to reference any other classes Hopefully this jives with what other people understand to be the "ideal" we're going for.
        Hide
        Todd Lipcon added a comment -

        Here's a patch which does the following:

        • FSImage no longer has a reference to FSNamesystem. Instead, the FSNamesystem is passed in as a parameter to save/load.
        • FSDirectory no longer is responsible for creating FSImage, and no longer goes through FSImage to get FSNamesystem. I wasn't able to completely remove the fsImage member in FSDirectory yet, but that's the next logical step.
        • FSDirectory no longer coordinates the loading of the image at startup. This code is in a new static method FSNamesystem.loadFromDisk which makes a little more sense
        • Removed some constructors from FSNamesystem that were made extraneous by the above work

        The tests I've run so far seem to be passing.

        Show
        Todd Lipcon added a comment - Here's a patch which does the following: FSImage no longer has a reference to FSNamesystem. Instead, the FSNamesystem is passed in as a parameter to save/load. FSDirectory no longer is responsible for creating FSImage, and no longer goes through FSImage to get FSNamesystem. I wasn't able to completely remove the fsImage member in FSDirectory yet, but that's the next logical step. FSDirectory no longer coordinates the loading of the image at startup. This code is in a new static method FSNamesystem.loadFromDisk which makes a little more sense Removed some constructors from FSNamesystem that were made extraneous by the above work The tests I've run so far seem to be passing.
        Hide
        Todd Lipcon added a comment -

        A little further cleanup - no longer need to keep checkpointDirs around as state inside FSImage, since it's truly a load-time thing.

        Show
        Todd Lipcon added a comment - A little further cleanup - no longer need to keep checkpointDirs around as state inside FSImage, since it's truly a load-time thing.
        Hide
        Aaron T. Myers added a comment -

        +1, patch looks good to me.

        One question: It seems like it wouldn't be too tough to also remove the reference to FSNamesystem from FSDirectory. Why not do that here? Feel free to say this should be done in a separate JIRA.

        Show
        Aaron T. Myers added a comment - +1, patch looks good to me. One question: It seems like it wouldn't be too tough to also remove the reference to FSNamesystem from FSDirectory . Why not do that here? Feel free to say this should be done in a separate JIRA.
        Hide
        Aaron T. Myers added a comment -

        Exact same patch as Todd's, but generated with the --no-prefix option.

        Show
        Aaron T. Myers added a comment - Exact same patch as Todd's, but generated with the --no-prefix option.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        > FSImage no longer has a reference to FSNamesystem. Instead, the FSNamesystem is passed in as a parameter to save/load.

        Should FSImage.namesystem be removed?

        Show
        Tsz Wo Nicholas Sze added a comment - > FSImage no longer has a reference to FSNamesystem. Instead, the FSNamesystem is passed in as a parameter to save/load. Should FSImage.namesystem be removed?
        Hide
        Suresh Srinivas added a comment -

        container for BlockManager, DatanodeManager, DelegationTokens, LeaseManager, etc

        Why not container for FSDirectory, FSImage, FSEditlog, as they are very closely related to namespace.

        Should NNStorage belong to NameNode?

        Show
        Suresh Srinivas added a comment - container for BlockManager, DatanodeManager, DelegationTokens, LeaseManager, etc Why not container for FSDirectory, FSImage, FSEditlog, as they are very closely related to namespace. Should NNStorage belong to NameNode?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12489921/hdfs-2223-3.txt
        against trunk revision 1155998.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 15 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1063//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12489921/hdfs-2223-3.txt against trunk revision 1155998. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1063//console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        Rebased on trunk

        Show
        Todd Lipcon added a comment - Rebased on trunk
        Hide
        Todd Lipcon added a comment -

        Should FSImage.namesystem be removed?

        Yes - it is removed in this patch. I wasn't able to easily remove it from BackupImage at this point, without changing a bunch of the BN around. So, I left it in the subclass only.

        Why not container for FSDirectory, FSImage, FSEditlog, as they are very closely related to namespace.

        Should NNStorage belong to NameNode?

        I think FSEditLog still belongs below FSImage, as it's part of the "image on disk" in a sense, and their lifecycle is tightly related.
        Moving NNStorage up into NameNode does make a certain amount of sense to me, but I didn't want to bite off too much in one patch.

        Do you think the changes in the attached patch are good, even if they're not a complete cleanup of everything? If so, let's commit this and then do some more work in another JIRA, just so each small improvement is easy to review?

        Show
        Todd Lipcon added a comment - Should FSImage.namesystem be removed? Yes - it is removed in this patch. I wasn't able to easily remove it from BackupImage at this point, without changing a bunch of the BN around. So, I left it in the subclass only. Why not container for FSDirectory, FSImage, FSEditlog, as they are very closely related to namespace. Should NNStorage belong to NameNode? I think FSEditLog still belongs below FSImage, as it's part of the "image on disk" in a sense, and their lifecycle is tightly related. Moving NNStorage up into NameNode does make a certain amount of sense to me, but I didn't want to bite off too much in one patch. Do you think the changes in the attached patch are good, even if they're not a complete cleanup of everything? If so, let's commit this and then do some more work in another JIRA, just so each small improvement is easy to review?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12490055/hdfs-2223-4.txt
        against trunk revision 1155998.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 15 new or modified tests.

        -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
        org.apache.hadoop.hdfs.server.common.TestDistributedUpgrade
        org.apache.hadoop.hdfs.server.namenode.TestCheckpoint
        org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
        org.apache.hadoop.hdfs.server.namenode.TestValidateConfigurationSettings
        org.apache.hadoop.hdfs.TestHDFSServerPorts
        org.apache.hadoop.tools.TestJMXGet

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1079//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1079//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1079//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12490055/hdfs-2223-4.txt against trunk revision 1155998. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy org.apache.hadoop.hdfs.server.common.TestDistributedUpgrade org.apache.hadoop.hdfs.server.namenode.TestCheckpoint org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark org.apache.hadoop.hdfs.server.namenode.TestValidateConfigurationSettings org.apache.hadoop.hdfs.TestHDFSServerPorts org.apache.hadoop.tools.TestJMXGet +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1079//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1079//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1079//console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        Looks like at least one of those test issues is legit. Looking into it.

        Show
        Todd Lipcon added a comment - Looks like at least one of those test issues is legit. Looking into it.
        Hide
        Todd Lipcon added a comment -

        Diffs since previous patch:

        • only register the FSN MBean when the NN is activated, not when it's constructed
        • sets storage.upgradeManager earlier during startup to fix TestDistributedUpgrade
        • improves TestDistributedUpgrade to be easier to debug
        • fixes javadoc warnings (trivial)
        Show
        Todd Lipcon added a comment - Diffs since previous patch: only register the FSN MBean when the NN is activated, not when it's constructed sets storage.upgradeManager earlier during startup to fix TestDistributedUpgrade improves TestDistributedUpgrade to be easier to debug fixes javadoc warnings (trivial)
        Hide
        Suresh Srinivas added a comment -

        Rebased patch to work with new trunk.

        Show
        Suresh Srinivas added a comment - Rebased patch to work with new trunk.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12491802/hdfs-2223-6.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 18 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:

        org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1160//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1160//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1160//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12491802/hdfs-2223-6.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1160//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1160//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1160//console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        I ran TestHost2NodesMap locally and it passed - it's been failing on hudson lately, tracked by a different JIRA.

        Suresh, can you give a +1 if the patch looks good? Thanks!

        Show
        Todd Lipcon added a comment - I ran TestHost2NodesMap locally and it passed - it's been failing on hudson lately, tracked by a different JIRA. Suresh, can you give a +1 if the patch looks good? Thanks!
        Hide
        Suresh Srinivas added a comment -

        Comments:

        1. FSImage creating FSNamesystem() does not seem right. Especially when FSNamesystem#loadFSImage() calls FSImage#format(), which then creates a new FSNamesystem. It would be good to create FSNamsystem and pass it to FSImage in #format().
        2. BackupImage#journal() BackupImage#convergeJournalSpool() could take FSNamesystem as additional argument. We could remove FSNamesystem member variable. BTW I do not see any call to BackupImage#saveCheckPoint(). These changes can be done in a separate jira.
        3. Question unrelated to this patch: why is the expectation that configuration have both DFS_NAMENODE_CHECKPOINT_EDITS_DIR_KEY and DFS_NAMENODE_CHECKPOINT_DIR_KEY set, in FSImage#doImportCheckpoint()?
        4. Please add javadoc to FSNamesystem#loadFromDisk()
        5. Minor: FSNamesystem#initialize() - systemStart variable is no longer used
        6. Minor: Unnecessary change in NNStorage.java that pushes code beyond 80 columns
        Show
        Suresh Srinivas added a comment - Comments: FSImage creating FSNamesystem() does not seem right. Especially when FSNamesystem#loadFSImage() calls FSImage#format(), which then creates a new FSNamesystem. It would be good to create FSNamsystem and pass it to FSImage in #format(). BackupImage#journal() BackupImage#convergeJournalSpool() could take FSNamesystem as additional argument. We could remove FSNamesystem member variable. BTW I do not see any call to BackupImage#saveCheckPoint(). These changes can be done in a separate jira. Question unrelated to this patch: why is the expectation that configuration have both DFS_NAMENODE_CHECKPOINT_EDITS_DIR_KEY and DFS_NAMENODE_CHECKPOINT_DIR_KEY set, in FSImage#doImportCheckpoint()? Please add javadoc to FSNamesystem#loadFromDisk() Minor: FSNamesystem#initialize() - systemStart variable is no longer used Minor: Unnecessary change in NNStorage.java that pushes code beyond 80 columns
        Hide
        Todd Lipcon added a comment -

        Thanks for the comments, Suresh. I will work on a new patch.

        Show
        Todd Lipcon added a comment - Thanks for the comments, Suresh. I will work on a new patch.
        Hide
        Todd Lipcon added a comment -

        FSImage creating FSNamesystem() does not seem right. Especially when FSNamesystem#loadFSImage() calls FSImage#format(), which then creates a new FSNamesystem. It would be good to create FSNamsystem and pass it to FSImage in #format().

        Fixed. I had to also move the initialization of NNResourceChecker, since with this modification FSNamesystem was being created before FSImage. NNResourceChecker should probably move down into FSImage at some point, but I didn't want to broaden the scope of this JIRA further.

        BackupImage#journal() BackupImage#convergeJournalSpool() could take FSNamesystem as additional argument. We could remove FSNamesystem member variable. BTW I do not see any call to BackupImage#saveCheckPoint(). These changes can be done in a separate jira.

        Agreed this could be done separately.

        Question unrelated to this patch: why is the expectation that configuration have both DFS_NAMENODE_CHECKPOINT_EDITS_DIR_KEY and DFS_NAMENODE_CHECKPOINT_DIR_KEY set, in FSImage#doImportCheckpoint()?

        The dfs.namenode.checkpoint.edits.dir configuration defaults to {{$

        {dfs.namenode.checkpoint.dir}

        }}, so setting just the former without explicitly setting the latter is sufficient.

        Please add javadoc to FSNamesystem#loadFromDisk()

        Done

        Minor: FSNamesystem#initialize() - systemStart variable is no longer used

        It's still used from getStartTime() and SafeMode.leave.

        Minor: Unnecessary change in NNStorage.java that pushes code beyond 80 columns

        Oops, fixed.

        Show
        Todd Lipcon added a comment - FSImage creating FSNamesystem() does not seem right. Especially when FSNamesystem#loadFSImage() calls FSImage#format(), which then creates a new FSNamesystem. It would be good to create FSNamsystem and pass it to FSImage in #format(). Fixed. I had to also move the initialization of NNResourceChecker, since with this modification FSNamesystem was being created before FSImage. NNResourceChecker should probably move down into FSImage at some point, but I didn't want to broaden the scope of this JIRA further. BackupImage#journal() BackupImage#convergeJournalSpool() could take FSNamesystem as additional argument. We could remove FSNamesystem member variable. BTW I do not see any call to BackupImage#saveCheckPoint(). These changes can be done in a separate jira. Agreed this could be done separately. Question unrelated to this patch: why is the expectation that configuration have both DFS_NAMENODE_CHECKPOINT_EDITS_DIR_KEY and DFS_NAMENODE_CHECKPOINT_DIR_KEY set, in FSImage#doImportCheckpoint()? The dfs.namenode.checkpoint.edits.dir configuration defaults to {{$ {dfs.namenode.checkpoint.dir} }}, so setting just the former without explicitly setting the latter is sufficient. Please add javadoc to FSNamesystem#loadFromDisk() Done Minor: FSNamesystem#initialize() - systemStart variable is no longer used It's still used from getStartTime() and SafeMode.leave. Minor: Unnecessary change in NNStorage.java that pushes code beyond 80 columns Oops, fixed.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12492545/hdfs-2223-7.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 18 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1196//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12492545/hdfs-2223-7.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1196//console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        Rebased on trunk

        Show
        Todd Lipcon added a comment - Rebased on trunk
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12493075/hdfs-2223-8.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 18 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).

        -1 core tests. The patch failed these unit tests:

        org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1197//testReport/
        Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1197//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1197//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1197//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493075/hdfs-2223-8.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1197//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1197//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1197//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1197//console This message is automatically generated.
        Hide
        Suresh Srinivas added a comment -

        +1 for the latest patch with -1s from hudson addressed.

        Show
        Suresh Srinivas added a comment - +1 for the latest patch with -1s from hudson addressed.
        Hide
        Todd Lipcon added a comment -

        Looking into the -1s, I see that this is causing TestEditLogRace.testSaveNamespace to time out. It's a kind of messy situation – the FSN rwlock is deadlocking against its fairness policy:

        • saveNamespace acquires the read lock
        • spawns an image saver thread
        • another thread comes in to do mkdirs and is waiting on the write lock
        • the image saver thread calls getNamespaceInfo (new in this patch) and wants the read lock
          Fairness policy says that the image saver thread can't get the read lock, since someone is already waiting on the write lock. So, it hangs there. The writer hangs on the main thread, and the main thread is join()ing on the saver thread.

        I think the easiest solution is to unsynchronize getNamespaceInfo, since in practice none of those fields change after the FSN is initialized. I will upload a new patch.

        Show
        Todd Lipcon added a comment - Looking into the -1s, I see that this is causing TestEditLogRace.testSaveNamespace to time out. It's a kind of messy situation – the FSN rwlock is deadlocking against its fairness policy: saveNamespace acquires the read lock spawns an image saver thread another thread comes in to do mkdirs and is waiting on the write lock the image saver thread calls getNamespaceInfo (new in this patch) and wants the read lock Fairness policy says that the image saver thread can't get the read lock, since someone is already waiting on the write lock. So, it hangs there. The writer hangs on the main thread, and the main thread is join()ing on the saver thread. I think the easiest solution is to unsynchronize getNamespaceInfo, since in practice none of those fields change after the FSN is initialized. I will upload a new patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12493087/hdfs-2223-9.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 18 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).

        -1 core tests. The patch failed these unit tests:

        org.apache.hadoop.hdfs.TestDfsOverAvroRpc
        org.apache.hadoop.hdfs.server.namenode.TestBackupNode
        org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1200//testReport/
        Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1200//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1200//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1200//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493087/hdfs-2223-9.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestDfsOverAvroRpc org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1200//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1200//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1200//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1200//console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        The above tests and release audit warnings are known issues in trunk addressed by other JIRAs. Suresh, if the above comment about getNamespaceInfo synchronization makes sense, I will go ahead and commit.

        Show
        Todd Lipcon added a comment - The above tests and release audit warnings are known issues in trunk addressed by other JIRAs. Suresh, if the above comment about getNamespaceInfo synchronization makes sense, I will go ahead and commit.
        Hide
        Suresh Srinivas added a comment -

        I have some concerns about removing readLock() completely. While we do not expect the values to change, the visibility is not guaranteed without locks. How about the following changes:

        1. upgradeManager.getUpgradeVersion() is a synchronized method. So it does not required readLock()
        2. FSNamesystem#dir is non volatile - a thread accessing it could just see null stale value. Could you make it final and initialize it in the constructor. BTW I have also been thinking that instead of using initialize method, which prevents making some member vars non final, we should initialize them in the constructor it self.
        3. Can you please make the following final FSImage#storage, FSDirectory#fsImage
        Show
        Suresh Srinivas added a comment - I have some concerns about removing readLock() completely. While we do not expect the values to change, the visibility is not guaranteed without locks. How about the following changes: upgradeManager.getUpgradeVersion() is a synchronized method. So it does not required readLock() FSNamesystem#dir is non volatile - a thread accessing it could just see null stale value. Could you make it final and initialize it in the constructor. BTW I have also been thinking that instead of using initialize method, which prevents making some member vars non final, we should initialize them in the constructor it self. Can you please make the following final FSImage#storage, FSDirectory#fsImage
        Hide
        Suresh Srinivas added a comment -

        Thinking a bit more - cluster id, block pool ID etc. also will have unsynchronized access even with my suggestion.

        Show
        Suresh Srinivas added a comment - Thinking a bit more - cluster id, block pool ID etc. also will have unsynchronized access even with my suggestion.
        Hide
        Todd Lipcon added a comment -

        How about this: we can keep getNamespaceInfo with the readlock, and delegate inside the lock to getNamespaceInfoInternal(). From within saveNamespace, we can directly call the getNamespaceInfoInternal method, since we know that the outer coordinating thread of saveNamespace is holding the lock "on our behalf".

        Show
        Todd Lipcon added a comment - How about this: we can keep getNamespaceInfo with the readlock, and delegate inside the lock to getNamespaceInfoInternal(). From within saveNamespace, we can directly call the getNamespaceInfoInternal method, since we know that the outer coordinating thread of saveNamespace is holding the lock "on our behalf".
        Hide
        Todd Lipcon added a comment -

        Here's a new version of the patch, rebased on trunk, implementing the above idea.

        Show
        Todd Lipcon added a comment - Here's a new version of the patch, rebased on trunk, implementing the above idea.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12493506/hdfs-2223-10.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 18 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:

        org.apache.hadoop.hdfs.TestDfsOverAvroRpc
        org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap
        org.apache.hadoop.hdfs.server.datanode.TestReplicasMap

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1222//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1222//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1222//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493506/hdfs-2223-10.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestDfsOverAvroRpc org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap org.apache.hadoop.hdfs.server.datanode.TestReplicasMap +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1222//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1222//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1222//console This message is automatically generated.
        Hide
        Suresh Srinivas added a comment -

        How about this: we can keep getNamespaceInfo with the readlock, and delegate inside the lock to getNamespaceInfoInternal(). From within saveNamespace, we can directly call the getNamespaceInfoInternal method, since we know that the outer coordinating thread of saveNamespace is holding the lock "on our behalf".

        Nice idea. +1 for the patch.

        Show
        Suresh Srinivas added a comment - How about this: we can keep getNamespaceInfo with the readlock, and delegate inside the lock to getNamespaceInfoInternal(). From within saveNamespace, we can directly call the getNamespaceInfoInternal method, since we know that the outer coordinating thread of saveNamespace is holding the lock "on our behalf". Nice idea. +1 for the patch.
        Hide
        Todd Lipcon added a comment -

        Committed to trunk for now. We may want to merge this to 0.23 and/or HA soon as well, but will hold off on that for the time being. Thanks for reviewing, Suresh.

        Show
        Todd Lipcon added a comment - Committed to trunk for now. We may want to merge this to 0.23 and/or HA soon as well, but will hold off on that for the time being. Thanks for reviewing, Suresh.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #846 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/846/)
        HDFS-2223. Untangle depencencies between NN components. Contributed by Todd Lipcon.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166466
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestDistributedUpgrade.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/unit/org/apache/hadoop/hdfs/server/namenode/TestNNLeaseRecovery.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #846 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/846/ ) HDFS-2223 . Untangle depencencies between NN components. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166466 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestDistributedUpgrade.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/unit/org/apache/hadoop/hdfs/server/namenode/TestNNLeaseRecovery.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #923 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/923/)
        HDFS-2223. Untangle depencencies between NN components. Contributed by Todd Lipcon.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166466
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestDistributedUpgrade.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/unit/org/apache/hadoop/hdfs/server/namenode/TestNNLeaseRecovery.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #923 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/923/ ) HDFS-2223 . Untangle depencencies between NN components. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166466 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestDistributedUpgrade.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/unit/org/apache/hadoop/hdfs/server/namenode/TestNNLeaseRecovery.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #857 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/857/)
        HDFS-2223. Untangle depencencies between NN components. Contributed by Todd Lipcon.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166466
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestDistributedUpgrade.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/unit/org/apache/hadoop/hdfs/server/namenode/TestNNLeaseRecovery.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #857 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/857/ ) HDFS-2223 . Untangle depencencies between NN components. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166466 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestDistributedUpgrade.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/unit/org/apache/hadoop/hdfs/server/namenode/TestNNLeaseRecovery.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #787 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/787/)
        HDFS-2223. Untangle depencencies between NN components. Contributed by Todd Lipcon.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166466
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestDistributedUpgrade.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/unit/org/apache/hadoop/hdfs/server/namenode/TestNNLeaseRecovery.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #787 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/787/ ) HDFS-2223 . Untangle depencencies between NN components. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166466 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestDistributedUpgrade.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/unit/org/apache/hadoop/hdfs/server/namenode/TestNNLeaseRecovery.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #810 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/810/)
        HDFS-2223. Untangle depencencies between NN components. Contributed by Todd Lipcon.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166466
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestDistributedUpgrade.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/unit/org/apache/hadoop/hdfs/server/namenode/TestNNLeaseRecovery.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #810 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/810/ ) HDFS-2223 . Untangle depencencies between NN components. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166466 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestDistributedUpgrade.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/unit/org/apache/hadoop/hdfs/server/namenode/TestNNLeaseRecovery.java
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have merged this to 0.23.

        Show
        Tsz Wo Nicholas Sze added a comment - I have merged this to 0.23.

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development