Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1052

HDFS scalability with multiple namenodes

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.23.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HDFS currently uses a single namenode that limits scalability of the cluster. This jira proposes an architecture to scale the nameservice horizontally using multiple namenodes.

      1. Block pool proposal.pdf
        410 kB
        Suresh Srinivas
      2. Mulitple Namespaces5.pdf
        1.85 MB
        Sanjay Radia
      3. high-level-design.pdf
        620 kB
        Suresh Srinivas
      4. HDFS-1052.patch
        968 kB
        Suresh Srinivas
      5. HDFS-1052.3.patch
        1.09 MB
        Suresh Srinivas
      6. HDFS-1052.4.patch
        1.09 MB
        Suresh Srinivas
      7. HDFS-1052.5.patch
        1.09 MB
        Suresh Srinivas
      8. HDFS-1052.6.patch
        1.10 MB
        Suresh Srinivas

        Issue Links

        1.
        HDFS federation: Add BlockPoolID to block Sub-task Resolved Suresh Srinivas
         
        2.
        HDFS federation: propose ClusterID and BlockPoolID format Sub-task Resolved Tanping Wang
         
        3.
        HDFS Federation: modify -format option for namenode to generated new blockpool id and accept newcluster Sub-task Resolved Boris Shkolnik
         
        4.
        HDFS federation: Storage directory of VERSION(/ID) file Sub-task Resolved Unassigned
         
        5.
        HDFS federation: Upgrade and rolling back of Federation Sub-task Resolved Unassigned
         
        6.
        HDFS federation : fix unit test cases Sub-task Resolved Unassigned
         
        7.
        HDFS federation : add cluster ID and block pool ID into Name node web UI Sub-task Resolved Tanping Wang
         
        8.
        HDFS Federation: Convert single threaded DataNode into per BlockPool thread model. Sub-task Resolved Boris Shkolnik
         
        9.
        HDFS federation: Introduce block pool ID into FSDatasetInterface Sub-task Resolved Suresh Srinivas
         
        10.
        HDFS Federation: DataNode.handleDiskError needs to inform ALL namenodes if a disk failed Sub-task Resolved Boris Shkolnik
         
        11.
        HDFS Federation: Add block pool management to FSDataset. Sub-task Resolved Suresh Srinivas
         
        12.
        HDFS Federation: Datanode fields that are no longer used should be removed Sub-task Resolved Boris Shkolnik
         
        13.
        HDFS Federation: add Datanode.getDNRegistration(String bpid) method Sub-task Resolved Boris Shkolnik
         
        14.
        HDFS Federation: remove namenode argument from DataNode constructor Sub-task Resolved Boris Shkolnik
         
        15.
        HDFS Federation: DatanodeCommand Finalize sent from namenode to datanode must include block pool ID Sub-task Resolved Suresh Srinivas
         
        16.
        HDFS Federation: MiniDFSCluster#waitActive() waits for ever in federation Sub-task Resolved Suresh Srinivas
         
        17.
        HDFS-Federation: Add support for multiple namenodes in MiniDFSCluster Sub-task Resolved Suresh Srinivas
         
        18.
        HDFS Federation: Fix TestDFSUpgrade and TestDFSRollback failures. Sub-task Resolved Suresh Srinivas
         
        19.
        HDFS Federation: Tests that corrupt block files fail due to changed block pool file path in federation Sub-task Resolved Suresh Srinivas
         
        20.
        HDFS Federation: Datanode doesn't start with two namenodes Sub-task Resolved Boris Shkolnik
         
        21.
        Federation: Datanode needs to send block pool usage information in registration Sub-task Resolved Suresh Srinivas
         
        22.
        Federation: Datanode servlets need information about the namenode to service the request Sub-task Resolved Suresh Srinivas
         
        23.
        Federation: FSDataset in Datanode should be created after initial handshake with namenode Sub-task Resolved Jitendra Nath Pandey
         
        24.
        Federation: Multiple namenode configuration Sub-task Resolved Jitendra Nath Pandey
         
        25.
        Federation: Datanode command to refresh namenode list at the datanode. Sub-task Resolved Jitendra Nath Pandey
         
        26.
        Hdfs Federation: Remove unnecessary TODO:FEDERATION comments. Sub-task Resolved Jitendra Nath Pandey
         
        27.
        HDFS Federation: remove dnRegistration from Datanode Sub-task Resolved Boris Shkolnik
         
        28.
        HDFS Federation: shutdown in DataNode should be able to shutdown individual BP threads as well as the whole DN Sub-task Resolved Boris Shkolnik
         
        29.
        HDFS Federation: refactor stopDatanode(name) to work with multiple Block Pools Sub-task Resolved Boris Shkolnik
         
        30.
        Federation: Datanode changes to track block token secret per namenode. Sub-task Resolved Suresh Srinivas
         
        31.
        HDFS Federation: BPOfferService exits after the first iteration of the loop Sub-task Resolved Tanping Wang
         
        32.
        HDFS Federation: data node storage structure changes and introduce block pool storage for Federation. Sub-task Resolved Tanping Wang
         
        33.
        Federation: Block received is sent with invalid DatanodeRegistration Sub-task Resolved Tanping Wang
         
        34.
        Federation: Only DataStorage must be locked using in_use.lock and no locks must be associated with BlockPoolStorage Sub-task Resolved Tanping Wang
         
        35.
        HDFS Federation : fix unit test case, TestReplication Sub-task Resolved Tanping Wang
         
        36.
        Federation: Tests fail due to null pointer exception in Datanode#shutdown() Sub-task Resolved Tanping Wang
         
        37.
        HDFS federation: fix unit test case, TestCheckpoint and TestDataNodeMXBean Sub-task Resolved Tanping Wang
         
        38.
        HDFS federation: Rename getPoolId() everywhere to getBlockPoolId() Sub-task Resolved Tanping Wang
         
        39.
        HDFS federation :Add new
 column
s of Block
Pool
Used 
and
 Block
Pool
Used
%
 into JSP Sub-task Resolved Tanping Wang
         
        40.
        HDFS Federation: Rename BlockPool class to BlockPoolSlice Sub-task Resolved Tanping Wang
         
        41.
        HDF federation: Fix TestFsck and TestListCorruptFileBlocks failures Sub-task Resolved Tanping Wang
         
        42.
        HDFS federation: Remove getBlockpool() for NameNodeMXBean in FSNameSystem. Sub-task Resolved Tanping Wang
         
        43.
        Federation: fix TestBalancer Sub-task Resolved Tsz Wo Nicholas Sze
         
        44.
        Federation: support per pool and per node policies in Balancer Sub-task Resolved Tsz Wo Nicholas Sze
         
        45.
        Federation: Change Balancer CLI for multiple namenodes and balancing policy Sub-task Resolved Tsz Wo Nicholas Sze
         
        46.
        Federation: Test Balancer With Multiple NameNodes Sub-task Resolved Tsz Wo Nicholas Sze
         
        47.
        Federation: Balancer cannot start with with multiple namenodes. Sub-task Resolved Tsz Wo Nicholas Sze
         
        48. Federation: Add more Balancer tests with federation setting Sub-task Open Tsz Wo Nicholas Sze
         
        49.
        Federation: Fix failures in fault injection tests, TestDiskError, TestDatanodeRestart and TestDFSTartupVersions. Sub-task Resolved Suresh Srinivas
         
        50.
        Hdfs Federation: Configuration for namenodes Sub-task Resolved Jitendra Nath Pandey
         
        51.
        Federation: TestDFSStorageStateRecovery fails Sub-task Resolved Suresh Srinivas
         
        52.
        Federation: SimulatedFSDataset changes to work with federation and multiple block pools Sub-task Resolved Suresh Srinivas
         
        53.
        HDFS federation: Fix testOIV and TestDatanodeUtils Sub-task Resolved Tanping Wang
         
        54.
        HDFS Federation: when build version doesn't match - datanode should wait (keep connecting) untill NN comes up with the right version Sub-task Resolved Boris Shkolnik
         
        55.
        HDFS federation: fix testBlockRecovery Sub-task Resolved Boris Shkolnik
         
        56.
        Hdfs Federation: TestOverReplicatedBlocks and TestWriteReplica failing Sub-task Resolved Jitendra Nath Pandey
         
        57.
        Federation: Bump up LAYOUT_VERSION in FSConstants.java for Federation before committing to trunk Sub-task Resolved Unassigned
         
        58.
        HDFS federation :TestHeartbeatHandling fails: wrong number of items in cmds array Sub-task Resolved Tanping Wang
         
        59.
        HDFS federation : Fix TestBackupNode and TestRefreshNamenodes test failure. Sub-task Resolved Tanping Wang
         
        60.
        HDFS federation: Improve start/stop scripts and add script to decommission datanodes Sub-task Closed Tanping Wang
         
        61.
        Federation: Add a tool that lists namenodes, secondary and backup nodes from the configuration file Sub-task Resolved Suresh Srinivas
         
        62.
        HDFS federation: Balancer command throws NullPointerException Sub-task Resolved Suresh Srinivas
         
        63.
        Hdfs Federation: TestFileAppend2, TestFileAppend3 and TestBlockTokenWithDFS failing Sub-task Resolved Jitendra Nath Pandey
         
        64.
        Hdfs Federation: Failure in browsing data on new namenodes Sub-task Resolved Jitendra Nath Pandey
         
        65.
        Federation: remove datanode's datanodeId TODOs Sub-task Resolved Boris Shkolnik
         
        66.
        Error "nnaddr url param is null" when clicking on a node from NN Live Node Link Sub-task Resolved Jitendra Nath Pandey
         
        67.
        HDFS Federation: create method for updating machine name in DataNode.java Sub-task Resolved Boris Shkolnik
         
        68.
        HDFS Federation: when looking up datanode we should use machineNmae (in testOverReplicatedBlocks) Sub-task Resolved Boris Shkolnik
         
        69.
        Hdfs Federation: Prevent DataBlockScanner from running in tight loop Sub-task Resolved Jitendra Nath Pandey
         
        70.
        HDFS Federation: refactor upgrade object in DataNode Sub-task Resolved Boris Shkolnik
         
        71.
        HDFS Federation: warning/error not generated when datanode sees inconsistent/different Cluster ID between namenodes Sub-task Resolved Boris Shkolnik
         
        72.
        Federation: Add decommission tests for federated namenodes Sub-task Resolved Suresh Srinivas
         
        73.
        Federation: FSDataset volumeMap access is not correctly synchronized Sub-task Resolved Suresh Srinivas
         
        74.
        HDFS Federation: MiniDFSCluster#waitActive() bug causes some tests to fail Sub-task Resolved Suresh Srinivas
         
        75.
        Federation: TestDFSRemove fails intermittently Sub-task Resolved Suresh Srinivas
         
        76.
        Federation: FSVolumeSet volumes is not synchronized correctly Sub-task Resolved Suresh Srinivas
         
        77.
        Hdfs Federation: Configuration for principal names should not be namenode specific. Sub-task Resolved Jitendra Nath Pandey
         
        78.
        HDFS Federation: Add flag to MiniDFSCluster to differentiate between two different modes-Federation and not. Sub-task Resolved Boris Shkolnik
         
        79.
        Federation: merge FSImage changes into FSImage+NNStorage refactoring in trunk Sub-task Resolved Suresh Srinivas
         
        80.
        Federation: Update the layout version for federation changes Sub-task Resolved Suresh Srinivas
         
        81.
        Federation: Add new layout version to offline image/edits viewer Sub-task Resolved Suresh Srinivas
         
        82.
        Federation: fix fault injection test build failure Sub-task Resolved Suresh Srinivas
         
        83.
        Hdfs Federation: TestFileAppend3 fails intermittently. Sub-task Resolved Jitendra Nath Pandey
         
        84.
        HDFS Federation: TestBackupNode Fails on Federation branch Sub-task Resolved Boris Shkolnik
         
        85.
        Hdfs Federation: TestListCorruptFileBlocks failing in federation branch. Sub-task Resolved Jitendra Nath Pandey
         
        86.
        Federation HDFS: testFsck fails Sub-task Resolved Boris Shkolnik
         
        87.
        Hdfs Federation: The BPOfferService must always connect to namenode as the login user. Sub-task Resolved Jitendra Nath Pandey
         
        88.
        Hdfs Federation: Add command to delete block pool directories from a datanode. Sub-task Resolved Jitendra Nath Pandey
         
        89.
        Federation: DatablockScanner should scan blocks for all the block pools. Sub-task Resolved Jitendra Nath Pandey
         
        90. Address all the federation TODOs Sub-task Open Suresh Srinivas
         
        91. Enable a single 2nn to checkpoint multiple nameservices Sub-task Open Unassigned
         

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Suresh Srinivas
              Reporter:
              Suresh Srinivas
            • Votes:
              1 Vote for this issue
              Watchers:
              73 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development