Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1052

HDFS scalability with multiple namenodes

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.22.0
    • 0.23.0
    • namenode
    • None
    • Reviewed

    Description

      HDFS currently uses a single namenode that limits scalability of the cluster. This jira proposes an architecture to scale the nameservice horizontally using multiple namenodes.

      Attachments

        1. HDFS-1052.6.patch
          1.10 MB
          Suresh Srinivas
        2. HDFS-1052.5.patch
          1.09 MB
          Suresh Srinivas
        3. HDFS-1052.4.patch
          1.09 MB
          Suresh Srinivas
        4. HDFS-1052.3.patch
          1.09 MB
          Suresh Srinivas
        5. HDFS-1052.patch
          968 kB
          Suresh Srinivas
        6. high-level-design.pdf
          620 kB
          Suresh Srinivas
        7. Mulitple Namespaces5.pdf
          1.85 MB
          Sanjay Radia
        8. Block pool proposal.pdf
          410 kB
          Suresh Srinivas

        Issue Links

        incorporates

        Bug - A problem which impairs or prevents the functions of the product. HDFS-1419 Federation: Three test cases need minor modification after the new block id change

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved

        New Feature - A new feature of the product, which has yet to be developed. HDFS-1400 HDFS federation: Introduce block pool ID into DataTransferProtocol

        • Major - Major loss of function.
        • Resolved
        is blocked by

        Bug - A problem which impairs or prevents the functions of the product. HDFS-1888 MiniDFSCluster#corruptBlockOnDatanodes() access must be public for MapReduce contrib raid

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-2467 HDFS-1052 changes break the raid contrib module in MapReduce

        • Major - Major loss of function.
        • Closed
        is part of

        New Feature - A new feature of the product, which has yet to be developed. HDFS-1051 Umbrella Jira for Scaling the HDFS Name Service

        • Major - Major loss of function.
        • Open
        is related to

        New Feature - A new feature of the product, which has yet to be developed. HADOOP-7257 A client side mount table to give per-application/per-job file system view

        • Major - Major loss of function.
        • Closed
        relates to

        Bug - A problem which impairs or prevents the functions of the product. HDFS-6795 docs update for -format command for namenode for Federation ( HDFS-1394 )

        • Major - Major loss of function.
        • Resolved
        1.
        		 HDFS federation: Add BlockPoolID to block Sub-task Resolved Suresh Srinivas Actions
        2.
        		 HDFS federation: propose ClusterID and BlockPoolID format Sub-task Resolved Tanping Wang Actions
        3.
        		 HDFS Federation: modify -format option for namenode to generated new blockpool id and accept newcluster Sub-task Resolved Boris Shkolnik Actions
        4.
        		 HDFS federation: Storage directory of VERSION(/ID) file Sub-task Resolved Unassigned Actions
        5.
        		 HDFS federation: Upgrade and rolling back of Federation Sub-task Resolved Unassigned Actions
        6.
        		 HDFS federation : fix unit test cases Sub-task Resolved Unassigned Actions
        7.
        		 HDFS federation : add cluster ID and block pool ID into Name node web UI Sub-task Resolved Tanping Wang Actions
        8.
        		 HDFS Federation: Convert single threaded DataNode into per BlockPool thread model. Sub-task Resolved Boris Shkolnik Actions
        9.
        		 HDFS federation: Introduce block pool ID into FSDatasetInterface Sub-task Resolved Suresh Srinivas Actions
        10.
        		 HDFS Federation: DataNode.handleDiskError needs to inform ALL namenodes if a disk failed Sub-task Resolved Boris Shkolnik Actions
        11.
        		 HDFS Federation: Add block pool management to FSDataset. Sub-task Resolved Suresh Srinivas Actions
        12.
        		 HDFS Federation: Datanode fields that are no longer used should be removed Sub-task Resolved Boris Shkolnik Actions
        13.
        		 HDFS Federation: add Datanode.getDNRegistration(String bpid) method Sub-task Resolved Boris Shkolnik Actions
        14.
        		 HDFS Federation: remove namenode argument from DataNode constructor Sub-task Resolved Boris Shkolnik Actions
        15.
        		 HDFS Federation: DatanodeCommand Finalize sent from namenode to datanode must include block pool ID Sub-task Resolved Suresh Srinivas Actions
        16.
        		 HDFS Federation: MiniDFSCluster#waitActive() waits for ever in federation Sub-task Resolved Suresh Srinivas Actions
        17.
        		 HDFS-Federation: Add support for multiple namenodes in MiniDFSCluster Sub-task Resolved Suresh Srinivas Actions
        18.
        		 HDFS Federation: Fix TestDFSUpgrade and TestDFSRollback failures. Sub-task Resolved Suresh Srinivas Actions
        19.
        		 HDFS Federation: Tests that corrupt block files fail due to changed block pool file path in federation Sub-task Resolved Suresh Srinivas Actions
        20.
        		 HDFS Federation: Datanode doesn't start with two namenodes Sub-task Resolved Boris Shkolnik Actions
        21.
        		 Federation: Datanode needs to send block pool usage information in registration Sub-task Resolved Suresh Srinivas Actions
        22.
        		 Federation: Datanode servlets need information about the namenode to service the request Sub-task Resolved Suresh Srinivas Actions
        23.
        		 Federation: FSDataset in Datanode should be created after initial handshake with namenode Sub-task Resolved Jitendra Nath Pandey Actions
        24.
        		 Federation: Multiple namenode configuration Sub-task Resolved Jitendra Nath Pandey Actions
        25.
        		 Federation: Datanode command to refresh namenode list at the datanode. Sub-task Resolved Jitendra Nath Pandey Actions
        26.
        		 Hdfs Federation: Remove unnecessary TODO:FEDERATION comments. Sub-task Resolved Jitendra Nath Pandey Actions
        27.
        		 HDFS Federation: remove dnRegistration from Datanode Sub-task Resolved Boris Shkolnik Actions
        28.
        		 HDFS Federation: shutdown in DataNode should be able to shutdown individual BP threads as well as the whole DN Sub-task Resolved Boris Shkolnik Actions
        29.
        		 HDFS Federation: refactor stopDatanode(name) to work with multiple Block Pools Sub-task Resolved Boris Shkolnik Actions
        30.
        		 Federation: Datanode changes to track block token secret per namenode. Sub-task Resolved Suresh Srinivas Actions
        31.
        		 HDFS Federation: BPOfferService exits after the first iteration of the loop Sub-task Resolved Tanping Wang Actions
        32.
        		 HDFS Federation: data node storage structure changes and introduce block pool storage for Federation. Sub-task Resolved Tanping Wang Actions
        33.
        		 Federation: Block received is sent with invalid DatanodeRegistration Sub-task Resolved Tanping Wang Actions
        34.
        		 Federation: Only DataStorage must be locked using in_use.lock and no locks must be associated with BlockPoolStorage Sub-task Resolved Tanping Wang Actions
        35.
        		 HDFS Federation : fix unit test case, TestReplication Sub-task Resolved Tanping Wang Actions
        36.
        		 Federation: Tests fail due to null pointer exception in Datanode#shutdown() Sub-task Resolved Tanping Wang Actions
        37.
        		 HDFS federation: fix unit test case, TestCheckpoint and TestDataNodeMXBean Sub-task Resolved Tanping Wang Actions
        38.
        		 HDFS federation: Rename getPoolId() everywhere to getBlockPoolId() Sub-task Resolved Tanping Wang Actions
        39.
        		 HDFS federation :Add new
 column
s of Block
Pool
Used 
and
 Block
Pool
Used
%
 into JSP Sub-task Resolved Tanping Wang Actions
        40.
        		 HDFS Federation: Rename BlockPool class to BlockPoolSlice Sub-task Resolved Tanping Wang Actions
        41.
        		 HDF federation: Fix TestFsck and TestListCorruptFileBlocks failures Sub-task Resolved Tanping Wang Actions
        42.
        		 HDFS federation: Remove getBlockpool() for NameNodeMXBean in FSNameSystem. Sub-task Resolved Tanping Wang Actions
        43.
        		 Federation: fix TestBalancer Sub-task Resolved Tsz-wo Sze Actions
        44.
        		 Federation: support per pool and per node policies in Balancer Sub-task Resolved Tsz-wo Sze Actions
        45.
        		 Federation: Change Balancer CLI for multiple namenodes and balancing policy Sub-task Resolved Tsz-wo Sze Actions
        46.
        		 Federation: Test Balancer With Multiple NameNodes Sub-task Resolved Tsz-wo Sze Actions
        47.
        		 Federation: Balancer cannot start with with multiple namenodes. Sub-task Resolved Tsz-wo Sze Actions
        48.
        		 Federation: Add more Balancer tests with federation setting Sub-task Resolved Bharat Viswanadham Actions
        49.
        		 Federation: Fix failures in fault injection tests, TestDiskError, TestDatanodeRestart and TestDFSTartupVersions. Sub-task Resolved Suresh Srinivas Actions
        50.
        		 Hdfs Federation: Configuration for namenodes Sub-task Resolved Jitendra Nath Pandey Actions
        51.
        		 Federation: TestDFSStorageStateRecovery fails Sub-task Resolved Suresh Srinivas Actions
        52.
        		 Federation: SimulatedFSDataset changes to work with federation and multiple block pools Sub-task Resolved Suresh Srinivas Actions
        53.
        		 HDFS federation: Fix testOIV and TestDatanodeUtils Sub-task Resolved Tanping Wang Actions
        54.
        		 HDFS Federation: when build version doesn't match - datanode should wait (keep connecting) untill NN comes up with the right version Sub-task Resolved Boris Shkolnik Actions
        55.
        		 HDFS federation: fix testBlockRecovery Sub-task Resolved Boris Shkolnik Actions
        56.
        		 Hdfs Federation: TestOverReplicatedBlocks and TestWriteReplica failing Sub-task Resolved Jitendra Nath Pandey Actions
        57.
        		 Federation: Bump up LAYOUT_VERSION in FSConstants.java for Federation before committing to trunk Sub-task Resolved Unassigned Actions
        58.
        		 HDFS federation :TestHeartbeatHandling fails: wrong number of items in cmds array Sub-task Resolved Tanping Wang Actions
        59.
        		 HDFS federation : Fix TestBackupNode and TestRefreshNamenodes test failure. Sub-task Resolved Tanping Wang Actions
        60.
        		 HDFS federation: Improve start/stop scripts and add script to decommission datanodes Sub-task Closed Tanping Wang Actions
        61.
        		 Federation: Add a tool that lists namenodes, secondary and backup nodes from the configuration file Sub-task Resolved Suresh Srinivas Actions
        62.
        		 HDFS federation: Balancer command throws NullPointerException Sub-task Resolved Suresh Srinivas Actions
        63.
        		 Hdfs Federation: TestFileAppend2, TestFileAppend3 and TestBlockTokenWithDFS failing Sub-task Resolved Jitendra Nath Pandey Actions
        64.
        		 Hdfs Federation: Failure in browsing data on new namenodes Sub-task Resolved Jitendra Nath Pandey Actions
        65.
        		 Federation: remove datanode's datanodeId TODOs Sub-task Resolved Boris Shkolnik Actions
        66.
        		 Error "nnaddr url param is null" when clicking on a node from NN Live Node Link Sub-task Resolved Jitendra Nath Pandey Actions
        67.
        		 HDFS Federation: create method for updating machine name in DataNode.java Sub-task Resolved Boris Shkolnik Actions
        68.
        		 HDFS Federation: when looking up datanode we should use machineNmae (in testOverReplicatedBlocks) Sub-task Resolved Boris Shkolnik Actions
        69.
        		 Hdfs Federation: Prevent DataBlockScanner from running in tight loop Sub-task Resolved Jitendra Nath Pandey Actions
        70.
        		 HDFS Federation: refactor upgrade object in DataNode Sub-task Resolved Boris Shkolnik Actions
        71.
        		 HDFS Federation: warning/error not generated when datanode sees inconsistent/different Cluster ID between namenodes Sub-task Resolved Boris Shkolnik Actions
        72.
        		 Federation: Add decommission tests for federated namenodes Sub-task Resolved Suresh Srinivas Actions
        73.
        		 Federation: FSDataset volumeMap access is not correctly synchronized Sub-task Resolved Suresh Srinivas Actions
        74.
        		 HDFS Federation: MiniDFSCluster#waitActive() bug causes some tests to fail Sub-task Resolved Suresh Srinivas Actions
        75.
        		 Federation: TestDFSRemove fails intermittently Sub-task Resolved Suresh Srinivas Actions
        76.
        		 Federation: FSVolumeSet volumes is not synchronized correctly Sub-task Resolved Suresh Srinivas Actions
        77.
        		 Hdfs Federation: Configuration for principal names should not be namenode specific. Sub-task Resolved Jitendra Nath Pandey Actions
        78.
        		 HDFS Federation: Add flag to MiniDFSCluster to differentiate between two different modes-Federation and not. Sub-task Resolved Boris Shkolnik Actions
        79.
        		 Federation: merge FSImage changes into FSImage+NNStorage refactoring in trunk Sub-task Resolved Suresh Srinivas Actions
        80.
        		 Federation: Update the layout version for federation changes Sub-task Resolved Suresh Srinivas Actions
        81.
        		 Federation: Add new layout version to offline image/edits viewer Sub-task Resolved Suresh Srinivas Actions
        82.
        		 Federation: fix fault injection test build failure Sub-task Resolved Suresh Srinivas Actions
        83.
        		 Hdfs Federation: TestFileAppend3 fails intermittently. Sub-task Resolved Jitendra Nath Pandey Actions
        84.
        		 HDFS Federation: TestBackupNode Fails on Federation branch Sub-task Resolved Boris Shkolnik Actions
        85.
        		 Hdfs Federation: TestListCorruptFileBlocks failing in federation branch. Sub-task Resolved Jitendra Nath Pandey Actions
        86.
        		 Federation HDFS: testFsck fails Sub-task Resolved Boris Shkolnik Actions
        87.
        		 Hdfs Federation: The BPOfferService must always connect to namenode as the login user. Sub-task Resolved Jitendra Nath Pandey Actions
        88.
        		 Hdfs Federation: Add command to delete block pool directories from a datanode. Sub-task Resolved Jitendra Nath Pandey Actions
        89.
        		 Federation: DatablockScanner should scan blocks for all the block pools. Sub-task Resolved Jitendra Nath Pandey Actions
        90.
        		 Address all the federation TODOs Sub-task Open Suresh Srinivas Actions
        91.
        		 Enable a single 2nn to checkpoint multiple nameservices Sub-task Open Unassigned Actions

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            sureshms Suresh Srinivas Assign to me
            sureshms Suresh Srinivas
            Votes:
            1 Vote for this issue
            Watchers:
            76 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment