Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1052

HDFS scalability with multiple namenodes

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.22.0
    • 0.23.0
    • namenode
    • None
    • Reviewed

    Description

      HDFS currently uses a single namenode that limits scalability of the cluster. This jira proposes an architecture to scale the nameservice horizontally using multiple namenodes.

      Attachments

        1. Mulitple Namespaces5.pdf
          1.85 MB
          Sanjay Radia
        2. high-level-design.pdf
          620 kB
          Suresh Srinivas
        3. HDFS-1052.patch
          968 kB
          Suresh Srinivas
        4. HDFS-1052.6.patch
          1.10 MB
          Suresh Srinivas
        5. HDFS-1052.5.patch
          1.09 MB
          Suresh Srinivas
        6. HDFS-1052.4.patch
          1.09 MB
          Suresh Srinivas
        7. HDFS-1052.3.patch
          1.09 MB
          Suresh Srinivas
        8. Block pool proposal.pdf
          410 kB
          Suresh Srinivas

        Issue Links

        1.
        HDFS federation: Add BlockPoolID to block Sub-task Resolved Suresh Srinivas Actions
        2.
        HDFS federation: propose ClusterID and BlockPoolID format Sub-task Resolved Tanping Wang Actions
        3.
        HDFS Federation: modify -format option for namenode to generated new blockpool id and accept newcluster Sub-task Resolved Boris Shkolnik Actions
        4.
        HDFS federation: Storage directory of VERSION(/ID) file Sub-task Resolved Unassigned Actions
        5.
        HDFS federation: Upgrade and rolling back of Federation Sub-task Resolved Unassigned Actions
        6.
        HDFS federation : fix unit test cases Sub-task Resolved Unassigned Actions
        7.
        HDFS federation : add cluster ID and block pool ID into Name node web UI Sub-task Resolved Tanping Wang Actions
        8.
        HDFS Federation: Convert single threaded DataNode into per BlockPool thread model. Sub-task Resolved Boris Shkolnik Actions
        9.
        HDFS federation: Introduce block pool ID into FSDatasetInterface Sub-task Resolved Suresh Srinivas Actions
        10.
        HDFS Federation: DataNode.handleDiskError needs to inform ALL namenodes if a disk failed Sub-task Resolved Boris Shkolnik Actions
        11.
        HDFS Federation: Add block pool management to FSDataset. Sub-task Resolved Suresh Srinivas Actions
        12.
        HDFS Federation: Datanode fields that are no longer used should be removed Sub-task Resolved Boris Shkolnik Actions
        13.
        HDFS Federation: add Datanode.getDNRegistration(String bpid) method Sub-task Resolved Boris Shkolnik Actions
        14.
        HDFS Federation: remove namenode argument from DataNode constructor Sub-task Resolved Boris Shkolnik Actions
        15.
        HDFS Federation: DatanodeCommand Finalize sent from namenode to datanode must include block pool ID Sub-task Resolved Suresh Srinivas Actions
        16.
        HDFS Federation: MiniDFSCluster#waitActive() waits for ever in federation Sub-task Resolved Suresh Srinivas Actions
        17.
        HDFS-Federation: Add support for multiple namenodes in MiniDFSCluster Sub-task Resolved Suresh Srinivas Actions
        18.
        HDFS Federation: Fix TestDFSUpgrade and TestDFSRollback failures. Sub-task Resolved Suresh Srinivas Actions
        19.
        HDFS Federation: Tests that corrupt block files fail due to changed block pool file path in federation Sub-task Resolved Suresh Srinivas Actions
        20.
        HDFS Federation: Datanode doesn't start with two namenodes Sub-task Resolved Boris Shkolnik Actions
        21.
        Federation: Datanode needs to send block pool usage information in registration Sub-task Resolved Suresh Srinivas Actions
        22.
        Federation: Datanode servlets need information about the namenode to service the request Sub-task Resolved Suresh Srinivas Actions
        23.
        Federation: FSDataset in Datanode should be created after initial handshake with namenode Sub-task Resolved Jitendra Nath Pandey Actions
        24.
        Federation: Multiple namenode configuration Sub-task Resolved Jitendra Nath Pandey Actions
        25.
        Federation: Datanode command to refresh namenode list at the datanode. Sub-task Resolved Jitendra Nath Pandey Actions
        26.
        Hdfs Federation: Remove unnecessary TODO:FEDERATION comments. Sub-task Resolved Jitendra Nath Pandey Actions
        27.
        HDFS Federation: remove dnRegistration from Datanode Sub-task Resolved Boris Shkolnik Actions
        28.
        HDFS Federation: shutdown in DataNode should be able to shutdown individual BP threads as well as the whole DN Sub-task Resolved Boris Shkolnik Actions
        29.
        HDFS Federation: refactor stopDatanode(name) to work with multiple Block Pools Sub-task Resolved Boris Shkolnik Actions
        30.
        Federation: Datanode changes to track block token secret per namenode. Sub-task Resolved Suresh Srinivas Actions
        31.
        HDFS Federation: BPOfferService exits after the first iteration of the loop Sub-task Resolved Tanping Wang Actions
        32.
        HDFS Federation: data node storage structure changes and introduce block pool storage for Federation. Sub-task Resolved Tanping Wang Actions
        33.
        Federation: Block received is sent with invalid DatanodeRegistration Sub-task Resolved Tanping Wang Actions
        34.
        Federation: Only DataStorage must be locked using in_use.lock and no locks must be associated with BlockPoolStorage Sub-task Resolved Tanping Wang Actions
        35.
        HDFS Federation : fix unit test case, TestReplication Sub-task Resolved Tanping Wang Actions
        36.
        Federation: Tests fail due to null pointer exception in Datanode#shutdown() Sub-task Resolved Tanping Wang Actions
        37.
        HDFS federation: fix unit test case, TestCheckpoint and TestDataNodeMXBean Sub-task Resolved Tanping Wang Actions
        38.
        HDFS federation: Rename getPoolId() everywhere to getBlockPoolId() Sub-task Resolved Tanping Wang Actions
        39.
        HDFS federation :Add new
 column
s of Block
Pool
Used 
and
 Block
Pool
Used
%
 into JSP Sub-task Resolved Tanping Wang Actions
        40.
        HDFS Federation: Rename BlockPool class to BlockPoolSlice Sub-task Resolved Tanping Wang Actions
        41.
        HDF federation: Fix TestFsck and TestListCorruptFileBlocks failures Sub-task Resolved Tanping Wang Actions
        42.
        HDFS federation: Remove getBlockpool() for NameNodeMXBean in FSNameSystem. Sub-task Resolved Tanping Wang Actions
        43.
        Federation: fix TestBalancer Sub-task Resolved Tsz-wo Sze Actions
        44.
        Federation: support per pool and per node policies in Balancer Sub-task Resolved Tsz-wo Sze Actions
        45.
        Federation: Change Balancer CLI for multiple namenodes and balancing policy Sub-task Resolved Tsz-wo Sze Actions
        46.
        Federation: Test Balancer With Multiple NameNodes Sub-task Resolved Tsz-wo Sze Actions
        47.
        Federation: Balancer cannot start with with multiple namenodes. Sub-task Resolved Tsz-wo Sze Actions
        48.
        Federation: Add more Balancer tests with federation setting Sub-task Resolved Bharat Viswanadham Actions
        49.
        Federation: Fix failures in fault injection tests, TestDiskError, TestDatanodeRestart and TestDFSTartupVersions. Sub-task Resolved Suresh Srinivas Actions
        50.
        Hdfs Federation: Configuration for namenodes Sub-task Resolved Jitendra Nath Pandey Actions
        51.
        Federation: TestDFSStorageStateRecovery fails Sub-task Resolved Suresh Srinivas Actions
        52.
        Federation: SimulatedFSDataset changes to work with federation and multiple block pools Sub-task Resolved Suresh Srinivas Actions
        53.
        HDFS federation: Fix testOIV and TestDatanodeUtils Sub-task Resolved Tanping Wang Actions
        54.
        HDFS Federation: when build version doesn't match - datanode should wait (keep connecting) untill NN comes up with the right version Sub-task Resolved Boris Shkolnik Actions
        55.
        HDFS federation: fix testBlockRecovery Sub-task Resolved Boris Shkolnik Actions
        56.
        Hdfs Federation: TestOverReplicatedBlocks and TestWriteReplica failing Sub-task Resolved Jitendra Nath Pandey Actions
        57.
        Federation: Bump up LAYOUT_VERSION in FSConstants.java for Federation before committing to trunk Sub-task Resolved Unassigned Actions
        58.
        HDFS federation :TestHeartbeatHandling fails: wrong number of items in cmds array Sub-task Resolved Tanping Wang Actions
        59.
        HDFS federation : Fix TestBackupNode and TestRefreshNamenodes test failure. Sub-task Resolved Tanping Wang Actions
        60.
        HDFS federation: Improve start/stop scripts and add script to decommission datanodes Sub-task Closed Tanping Wang Actions
        61.
        Federation: Add a tool that lists namenodes, secondary and backup nodes from the configuration file Sub-task Resolved Suresh Srinivas Actions
        62.
        HDFS federation: Balancer command throws NullPointerException Sub-task Resolved Suresh Srinivas Actions
        63.
        Hdfs Federation: TestFileAppend2, TestFileAppend3 and TestBlockTokenWithDFS failing Sub-task Resolved Jitendra Nath Pandey Actions
        64.
        Hdfs Federation: Failure in browsing data on new namenodes Sub-task Resolved Jitendra Nath Pandey Actions
        65.
        Federation: remove datanode's datanodeId TODOs Sub-task Resolved Boris Shkolnik Actions
        66.
        Error "nnaddr url param is null" when clicking on a node from NN Live Node Link Sub-task Resolved Jitendra Nath Pandey Actions
        67.
        HDFS Federation: create method for updating machine name in DataNode.java Sub-task Resolved Boris Shkolnik Actions
        68.
        HDFS Federation: when looking up datanode we should use machineNmae (in testOverReplicatedBlocks) Sub-task Resolved Boris Shkolnik Actions
        69.
        Hdfs Federation: Prevent DataBlockScanner from running in tight loop Sub-task Resolved Jitendra Nath Pandey Actions
        70.
        HDFS Federation: refactor upgrade object in DataNode Sub-task Resolved Boris Shkolnik Actions
        71.
        HDFS Federation: warning/error not generated when datanode sees inconsistent/different Cluster ID between namenodes Sub-task Resolved Boris Shkolnik Actions
        72.
        Federation: Add decommission tests for federated namenodes Sub-task Resolved Suresh Srinivas Actions
        73.
        Federation: FSDataset volumeMap access is not correctly synchronized Sub-task Resolved Suresh Srinivas Actions
        74.
        HDFS Federation: MiniDFSCluster#waitActive() bug causes some tests to fail Sub-task Resolved Suresh Srinivas Actions
        75.
        Federation: TestDFSRemove fails intermittently Sub-task Resolved Suresh Srinivas Actions
        76.
        Federation: FSVolumeSet volumes is not synchronized correctly Sub-task Resolved Suresh Srinivas Actions
        77.
        Hdfs Federation: Configuration for principal names should not be namenode specific. Sub-task Resolved Jitendra Nath Pandey Actions
        78.
        HDFS Federation: Add flag to MiniDFSCluster to differentiate between two different modes-Federation and not. Sub-task Resolved Boris Shkolnik Actions
        79.
        Federation: merge FSImage changes into FSImage+NNStorage refactoring in trunk Sub-task Resolved Suresh Srinivas Actions
        80.
        Federation: Update the layout version for federation changes Sub-task Resolved Suresh Srinivas Actions
        81.
        Federation: Add new layout version to offline image/edits viewer Sub-task Resolved Suresh Srinivas Actions
        82.
        Federation: fix fault injection test build failure Sub-task Resolved Suresh Srinivas Actions
        83.
        Hdfs Federation: TestFileAppend3 fails intermittently. Sub-task Resolved Jitendra Nath Pandey Actions
        84.
        HDFS Federation: TestBackupNode Fails on Federation branch Sub-task Resolved Boris Shkolnik Actions
        85.
        Hdfs Federation: TestListCorruptFileBlocks failing in federation branch. Sub-task Resolved Jitendra Nath Pandey Actions
        86.
        Federation HDFS: testFsck fails Sub-task Resolved Boris Shkolnik Actions
        87.
        Hdfs Federation: The BPOfferService must always connect to namenode as the login user. Sub-task Resolved Jitendra Nath Pandey Actions
        88.
        Hdfs Federation: Add command to delete block pool directories from a datanode. Sub-task Resolved Jitendra Nath Pandey Actions
        89.
        Federation: DatablockScanner should scan blocks for all the block pools. Sub-task Resolved Jitendra Nath Pandey Actions
        90.
        Address all the federation TODOs Sub-task Open Suresh Srinivas Actions
        91.
        Enable a single 2nn to checkpoint multiple nameservices Sub-task Open Unassigned Actions

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            sureshms Suresh Srinivas
            sureshms Suresh Srinivas
            Votes:
            1 Vote for this issue
            Watchers:
            82 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment