Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1052

HDFS scalability with multiple namenodes

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.23.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HDFS currently uses a single namenode that limits scalability of the cluster. This jira proposes an architecture to scale the nameservice horizontally using multiple namenodes.

        Attachments

        1. HDFS-1052.6.patch
          1.10 MB
          Suresh Srinivas
        2. HDFS-1052.5.patch
          1.09 MB
          Suresh Srinivas
        3. HDFS-1052.4.patch
          1.09 MB
          Suresh Srinivas
        4. HDFS-1052.3.patch
          1.09 MB
          Suresh Srinivas
        5. HDFS-1052.patch
          968 kB
          Suresh Srinivas
        6. high-level-design.pdf
          620 kB
          Suresh Srinivas
        7. Mulitple Namespaces5.pdf
          1.85 MB
          Sanjay Radia
        8. Block pool proposal.pdf
          410 kB
          Suresh Srinivas

          Issue Links

          1.
          HDFS federation: Add BlockPoolID to block Sub-task Resolved Suresh Srinivas
          2.
          HDFS federation: propose ClusterID and BlockPoolID format Sub-task Resolved Tanping Wang
          3.
          HDFS Federation: modify -format option for namenode to generated new blockpool id and accept newcluster Sub-task Resolved Boris Shkolnik
          4.
          HDFS federation: Storage directory of VERSION(/ID) file Sub-task Resolved Unassigned
          5.
          HDFS federation: Upgrade and rolling back of Federation Sub-task Resolved Unassigned
          6.
          HDFS federation : fix unit test cases Sub-task Resolved Unassigned
          7.
          HDFS federation : add cluster ID and block pool ID into Name node web UI Sub-task Resolved Tanping Wang
          8.
          HDFS Federation: Convert single threaded DataNode into per BlockPool thread model. Sub-task Resolved Boris Shkolnik
          9.
          HDFS federation: Introduce block pool ID into FSDatasetInterface Sub-task Resolved Suresh Srinivas
          10.
          HDFS Federation: DataNode.handleDiskError needs to inform ALL namenodes if a disk failed Sub-task Resolved Boris Shkolnik
          11.
          HDFS Federation: Add block pool management to FSDataset. Sub-task Resolved Suresh Srinivas
          12.
          HDFS Federation: Datanode fields that are no longer used should be removed Sub-task Resolved Boris Shkolnik
          13.
          HDFS Federation: add Datanode.getDNRegistration(String bpid) method Sub-task Resolved Boris Shkolnik
          14.
          HDFS Federation: remove namenode argument from DataNode constructor Sub-task Resolved Boris Shkolnik
          15.
          HDFS Federation: DatanodeCommand Finalize sent from namenode to datanode must include block pool ID Sub-task Resolved Suresh Srinivas
          16.
          HDFS Federation: MiniDFSCluster#waitActive() waits for ever in federation Sub-task Resolved Suresh Srinivas
          17.
          HDFS-Federation: Add support for multiple namenodes in MiniDFSCluster Sub-task Resolved Suresh Srinivas
          18.
          HDFS Federation: Fix TestDFSUpgrade and TestDFSRollback failures. Sub-task Resolved Suresh Srinivas
          19.
          HDFS Federation: Tests that corrupt block files fail due to changed block pool file path in federation Sub-task Resolved Suresh Srinivas
          20.
          HDFS Federation: Datanode doesn't start with two namenodes Sub-task Resolved Boris Shkolnik
          21.
          Federation: Datanode needs to send block pool usage information in registration Sub-task Resolved Suresh Srinivas
          22.
          Federation: Datanode servlets need information about the namenode to service the request Sub-task Resolved Suresh Srinivas
          23.
          Federation: FSDataset in Datanode should be created after initial handshake with namenode Sub-task Resolved Jitendra Nath Pandey
          24.
          Federation: Multiple namenode configuration Sub-task Resolved Jitendra Nath Pandey
          25.
          Federation: Datanode command to refresh namenode list at the datanode. Sub-task Resolved Jitendra Nath Pandey
          26.
          Hdfs Federation: Remove unnecessary TODO:FEDERATION comments. Sub-task Resolved Jitendra Nath Pandey
          27.
          HDFS Federation: remove dnRegistration from Datanode Sub-task Resolved Boris Shkolnik
          28.
          HDFS Federation: shutdown in DataNode should be able to shutdown individual BP threads as well as the whole DN Sub-task Resolved Boris Shkolnik
          29.
          HDFS Federation: refactor stopDatanode(name) to work with multiple Block Pools Sub-task Resolved Boris Shkolnik
          30.
          Federation: Datanode changes to track block token secret per namenode. Sub-task Resolved Suresh Srinivas
          31.
          HDFS Federation: BPOfferService exits after the first iteration of the loop Sub-task Resolved Tanping Wang
          32.
          HDFS Federation: data node storage structure changes and introduce block pool storage for Federation. Sub-task Resolved Tanping Wang
          33.
          Federation: Block received is sent with invalid DatanodeRegistration Sub-task Resolved Tanping Wang
          34.
          Federation: Only DataStorage must be locked using in_use.lock and no locks must be associated with BlockPoolStorage Sub-task Resolved Tanping Wang
          35.
          HDFS Federation : fix unit test case, TestReplication Sub-task Resolved Tanping Wang
          36.
          Federation: Tests fail due to null pointer exception in Datanode#shutdown() Sub-task Resolved Tanping Wang
          37.
          HDFS federation: fix unit test case, TestCheckpoint and TestDataNodeMXBean Sub-task Resolved Tanping Wang
          38.
          HDFS federation: Rename getPoolId() everywhere to getBlockPoolId() Sub-task Resolved Tanping Wang
          39.
          HDFS federation :Add new
 column
s of Block
Pool
Used 
and
 Block
Pool
Used
%
 into JSP Sub-task Resolved Tanping Wang
          40.
          HDFS Federation: Rename BlockPool class to BlockPoolSlice Sub-task Resolved Tanping Wang
          41.
          HDF federation: Fix TestFsck and TestListCorruptFileBlocks failures Sub-task Resolved Tanping Wang
          42.
          HDFS federation: Remove getBlockpool() for NameNodeMXBean in FSNameSystem. Sub-task Resolved Tanping Wang
          43.
          Federation: fix TestBalancer Sub-task Resolved Tsz Wo Nicholas Sze
          44.
          Federation: support per pool and per node policies in Balancer Sub-task Resolved Tsz Wo Nicholas Sze
          45.
          Federation: Change Balancer CLI for multiple namenodes and balancing policy Sub-task Resolved Tsz Wo Nicholas Sze
          46.
          Federation: Test Balancer With Multiple NameNodes Sub-task Resolved Tsz Wo Nicholas Sze
          47.
          Federation: Balancer cannot start with with multiple namenodes. Sub-task Resolved Tsz Wo Nicholas Sze
          48.
          Federation: Add more Balancer tests with federation setting Sub-task Resolved Bharat Viswanadham
          49.
          Federation: Fix failures in fault injection tests, TestDiskError, TestDatanodeRestart and TestDFSTartupVersions. Sub-task Resolved Suresh Srinivas
          50.
          Hdfs Federation: Configuration for namenodes Sub-task Resolved Jitendra Nath Pandey
          51.
          Federation: TestDFSStorageStateRecovery fails Sub-task Resolved Suresh Srinivas
          52.
          Federation: SimulatedFSDataset changes to work with federation and multiple block pools Sub-task Resolved Suresh Srinivas
          53.
          HDFS federation: Fix testOIV and TestDatanodeUtils Sub-task Resolved Tanping Wang
          54.
          HDFS Federation: when build version doesn't match - datanode should wait (keep connecting) untill NN comes up with the right version Sub-task Resolved Boris Shkolnik
          55.
          HDFS federation: fix testBlockRecovery Sub-task Resolved Boris Shkolnik
          56.
          Hdfs Federation: TestOverReplicatedBlocks and TestWriteReplica failing Sub-task Resolved Jitendra Nath Pandey
          57.
          Federation: Bump up LAYOUT_VERSION in FSConstants.java for Federation before committing to trunk Sub-task Resolved Unassigned
          58.
          HDFS federation :TestHeartbeatHandling fails: wrong number of items in cmds array Sub-task Resolved Tanping Wang
          59.
          HDFS federation : Fix TestBackupNode and TestRefreshNamenodes test failure. Sub-task Resolved Tanping Wang
          60.
          HDFS federation: Improve start/stop scripts and add script to decommission datanodes Sub-task Closed Tanping Wang
          61.
          Federation: Add a tool that lists namenodes, secondary and backup nodes from the configuration file Sub-task Resolved Suresh Srinivas
          62.
          HDFS federation: Balancer command throws NullPointerException Sub-task Resolved Suresh Srinivas
          63.
          Hdfs Federation: TestFileAppend2, TestFileAppend3 and TestBlockTokenWithDFS failing Sub-task Resolved Jitendra Nath Pandey
          64.
          Hdfs Federation: Failure in browsing data on new namenodes Sub-task Resolved Jitendra Nath Pandey
          65.
          Federation: remove datanode's datanodeId TODOs Sub-task Resolved Boris Shkolnik
          66.
          Error "nnaddr url param is null" when clicking on a node from NN Live Node Link Sub-task Resolved Jitendra Nath Pandey
          67.
          HDFS Federation: create method for updating machine name in DataNode.java Sub-task Resolved Boris Shkolnik
          68.
          HDFS Federation: when looking up datanode we should use machineNmae (in testOverReplicatedBlocks) Sub-task Resolved Boris Shkolnik
          69.
          Hdfs Federation: Prevent DataBlockScanner from running in tight loop Sub-task Resolved Jitendra Nath Pandey
          70.
          HDFS Federation: refactor upgrade object in DataNode Sub-task Resolved Boris Shkolnik
          71.
          HDFS Federation: warning/error not generated when datanode sees inconsistent/different Cluster ID between namenodes Sub-task Resolved Boris Shkolnik
          72.
          Federation: Add decommission tests for federated namenodes Sub-task Resolved Suresh Srinivas
          73.
          Federation: FSDataset volumeMap access is not correctly synchronized Sub-task Resolved Suresh Srinivas
          74.
          HDFS Federation: MiniDFSCluster#waitActive() bug causes some tests to fail Sub-task Resolved Suresh Srinivas
          75.
          Federation: TestDFSRemove fails intermittently Sub-task Resolved Suresh Srinivas
          76.
          Federation: FSVolumeSet volumes is not synchronized correctly Sub-task Resolved Suresh Srinivas
          77.
          Hdfs Federation: Configuration for principal names should not be namenode specific. Sub-task Resolved Jitendra Nath Pandey
          78.
          HDFS Federation: Add flag to MiniDFSCluster to differentiate between two different modes-Federation and not. Sub-task Resolved Boris Shkolnik
          79.
          Federation: merge FSImage changes into FSImage+NNStorage refactoring in trunk Sub-task Resolved Suresh Srinivas
          80.
          Federation: Update the layout version for federation changes Sub-task Resolved Suresh Srinivas
          81.
          Federation: Add new layout version to offline image/edits viewer Sub-task Resolved Suresh Srinivas
          82.
          Federation: fix fault injection test build failure Sub-task Resolved Suresh Srinivas
          83.
          Hdfs Federation: TestFileAppend3 fails intermittently. Sub-task Resolved Jitendra Nath Pandey
          84.
          HDFS Federation: TestBackupNode Fails on Federation branch Sub-task Resolved Boris Shkolnik
          85.
          Hdfs Federation: TestListCorruptFileBlocks failing in federation branch. Sub-task Resolved Jitendra Nath Pandey
          86.
          Federation HDFS: testFsck fails Sub-task Resolved Boris Shkolnik
          87.
          Hdfs Federation: The BPOfferService must always connect to namenode as the login user. Sub-task Resolved Jitendra Nath Pandey
          88.
          Hdfs Federation: Add command to delete block pool directories from a datanode. Sub-task Resolved Jitendra Nath Pandey
          89.
          Federation: DatablockScanner should scan blocks for all the block pools. Sub-task Resolved Jitendra Nath Pandey
          90.
          Address all the federation TODOs Sub-task Open Suresh Srinivas
          91.
          Enable a single 2nn to checkpoint multiple nameservices Sub-task Open Unassigned

            Activity

              People

              • Assignee:
                sureshms Suresh Srinivas
                Reporter:
                sureshms Suresh Srinivas
              • Votes:
                1 Vote for this issue
                Watchers:
                80 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: