Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1052

HDFS scalability with multiple namenodes

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.22.0
    • 0.23.0
    • namenode
    • None
    • Reviewed

    Description

      HDFS currently uses a single namenode that limits scalability of the cluster. This jira proposes an architecture to scale the nameservice horizontally using multiple namenodes.

      Attachments

        1. Block pool proposal.pdf
          410 kB
          Suresh Srinivas
        2. HDFS-1052.3.patch
          1.09 MB
          Suresh Srinivas
        3. HDFS-1052.4.patch
          1.09 MB
          Suresh Srinivas
        4. HDFS-1052.5.patch
          1.09 MB
          Suresh Srinivas
        5. HDFS-1052.6.patch
          1.10 MB
          Suresh Srinivas
        6. HDFS-1052.patch
          968 kB
          Suresh Srinivas
        7. high-level-design.pdf
          620 kB
          Suresh Srinivas
        8. Mulitple Namespaces5.pdf
          1.85 MB
          Sanjay Radia

        Issue Links

          incorporates

          Bug - A problem which impairs or prevents the functions of the product. HDFS-1419 Federation: Three test cases need minor modification after the new block id change

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HDFS-1400 HDFS federation: Introduce block pool ID into DataTransferProtocol

          • Major - Major loss of function.
          • Resolved
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. HDFS-1888 MiniDFSCluster#corruptBlockOnDatanodes() access must be public for MapReduce contrib raid

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-2467 HDFS-1052 changes break the raid contrib module in MapReduce

          • Major - Major loss of function.
          • Closed
          is part of

          New Feature - A new feature of the product, which has yet to be developed. HDFS-1051 Umbrella Jira for Scaling the HDFS Name Service

          • Major - Major loss of function.
          • Open
          is related to

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-7257 A client side mount table to give per-application/per-job file system view

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-6795 docs update for -format command for namenode for Federation ( HDFS-1394 )

          • Major - Major loss of function.
          • Resolved
          1.
          		 HDFS federation: Add BlockPoolID to block Sub-task Resolved Suresh Srinivas
          2.
          		 HDFS federation: propose ClusterID and BlockPoolID format Sub-task Resolved Tanping Wang
          3.
          		 HDFS Federation: modify -format option for namenode to generated new blockpool id and accept newcluster Sub-task Resolved Boris Shkolnik
          4.
          		 HDFS federation: Storage directory of VERSION(/ID) file Sub-task Resolved Unassigned
          5.
          		 HDFS federation: Upgrade and rolling back of Federation Sub-task Resolved Unassigned
          6.
          		 HDFS federation : fix unit test cases Sub-task Resolved Unassigned
          7.
          		 HDFS federation : add cluster ID and block pool ID into Name node web UI Sub-task Resolved Tanping Wang
          8.
          		 HDFS Federation: Convert single threaded DataNode into per BlockPool thread model. Sub-task Resolved Boris Shkolnik
          9.
          		 HDFS federation: Introduce block pool ID into FSDatasetInterface Sub-task Resolved Suresh Srinivas
          10.
          		 HDFS Federation: DataNode.handleDiskError needs to inform ALL namenodes if a disk failed Sub-task Resolved Boris Shkolnik
          11.
          		 HDFS Federation: Add block pool management to FSDataset. Sub-task Resolved Suresh Srinivas
          12.
          		 HDFS Federation: Datanode fields that are no longer used should be removed Sub-task Resolved Boris Shkolnik
          13.
          		 HDFS Federation: add Datanode.getDNRegistration(String bpid) method Sub-task Resolved Boris Shkolnik
          14.
          		 HDFS Federation: remove namenode argument from DataNode constructor Sub-task Resolved Boris Shkolnik
          15.
          		 HDFS Federation: DatanodeCommand Finalize sent from namenode to datanode must include block pool ID Sub-task Resolved Suresh Srinivas
          16.
          		 HDFS Federation: MiniDFSCluster#waitActive() waits for ever in federation Sub-task Resolved Suresh Srinivas
          17.
          		 HDFS-Federation: Add support for multiple namenodes in MiniDFSCluster Sub-task Resolved Suresh Srinivas
          18.
          		 HDFS Federation: Fix TestDFSUpgrade and TestDFSRollback failures. Sub-task Resolved Suresh Srinivas
          19.
          		 HDFS Federation: Tests that corrupt block files fail due to changed block pool file path in federation Sub-task Resolved Suresh Srinivas
          20.
          		 HDFS Federation: Datanode doesn't start with two namenodes Sub-task Resolved Boris Shkolnik
          21.
          		 Federation: Datanode needs to send block pool usage information in registration Sub-task Resolved Suresh Srinivas
          22.
          		 Federation: Datanode servlets need information about the namenode to service the request Sub-task Resolved Suresh Srinivas
          23.
          		 Federation: FSDataset in Datanode should be created after initial handshake with namenode Sub-task Resolved Jitendra Nath Pandey
          24.
          		 Federation: Multiple namenode configuration Sub-task Resolved Jitendra Nath Pandey
          25.
          		 Federation: Datanode command to refresh namenode list at the datanode. Sub-task Resolved Jitendra Nath Pandey
          26.
          		 Hdfs Federation: Remove unnecessary TODO:FEDERATION comments. Sub-task Resolved Jitendra Nath Pandey
          27.
          		 HDFS Federation: remove dnRegistration from Datanode Sub-task Resolved Boris Shkolnik
          28.
          		 HDFS Federation: shutdown in DataNode should be able to shutdown individual BP threads as well as the whole DN Sub-task Resolved Boris Shkolnik
          29.
          		 HDFS Federation: refactor stopDatanode(name) to work with multiple Block Pools Sub-task Resolved Boris Shkolnik
          30.
          		 Federation: Datanode changes to track block token secret per namenode. Sub-task Resolved Suresh Srinivas
          31.
          		 HDFS Federation: BPOfferService exits after the first iteration of the loop Sub-task Resolved Tanping Wang
          32.
          		 HDFS Federation: data node storage structure changes and introduce block pool storage for Federation. Sub-task Resolved Tanping Wang
          33.
          		 Federation: Block received is sent with invalid DatanodeRegistration Sub-task Resolved Tanping Wang
          34.
          		 Federation: Only DataStorage must be locked using in_use.lock and no locks must be associated with BlockPoolStorage Sub-task Resolved Tanping Wang
          35.
          		 HDFS Federation : fix unit test case, TestReplication Sub-task Resolved Tanping Wang
          36.
          		 Federation: Tests fail due to null pointer exception in Datanode#shutdown() Sub-task Resolved Tanping Wang
          37.
          		 HDFS federation: fix unit test case, TestCheckpoint and TestDataNodeMXBean Sub-task Resolved Tanping Wang
          38.
          		 HDFS federation: Rename getPoolId() everywhere to getBlockPoolId() Sub-task Resolved Tanping Wang
          39.
          		 HDFS federation :Add new
 column
s of Block
Pool
Used 
and
 Block
Pool
Used
%
 into JSP Sub-task Resolved Tanping Wang
          40.
          		 HDFS Federation: Rename BlockPool class to BlockPoolSlice Sub-task Resolved Tanping Wang
          41.
          		 HDF federation: Fix TestFsck and TestListCorruptFileBlocks failures Sub-task Resolved Tanping Wang
          42.
          		 HDFS federation: Remove getBlockpool() for NameNodeMXBean in FSNameSystem. Sub-task Resolved Tanping Wang
          43.
          		 Federation: fix TestBalancer Sub-task Resolved Tsz-wo Sze
          44.
          		 Federation: support per pool and per node policies in Balancer Sub-task Resolved Tsz-wo Sze
          45.
          		 Federation: Change Balancer CLI for multiple namenodes and balancing policy Sub-task Resolved Tsz-wo Sze
          46.
          		 Federation: Test Balancer With Multiple NameNodes Sub-task Resolved Tsz-wo Sze
          47.
          		 Federation: Balancer cannot start with with multiple namenodes. Sub-task Resolved Tsz-wo Sze
          48.
          		 Federation: Add more Balancer tests with federation setting Sub-task Resolved Bharat Viswanadham
          49.
          		 Federation: Fix failures in fault injection tests, TestDiskError, TestDatanodeRestart and TestDFSTartupVersions. Sub-task Resolved Suresh Srinivas
          50.
          		 Hdfs Federation: Configuration for namenodes Sub-task Resolved Jitendra Nath Pandey
          51.
          		 Federation: TestDFSStorageStateRecovery fails Sub-task Resolved Suresh Srinivas
          52.
          		 Federation: SimulatedFSDataset changes to work with federation and multiple block pools Sub-task Resolved Suresh Srinivas
          53.
          		 HDFS federation: Fix testOIV and TestDatanodeUtils Sub-task Resolved Tanping Wang
          54.
          		 HDFS Federation: when build version doesn't match - datanode should wait (keep connecting) untill NN comes up with the right version Sub-task Resolved Boris Shkolnik
          55.
          		 HDFS federation: fix testBlockRecovery Sub-task Resolved Boris Shkolnik
          56.
          		 Hdfs Federation: TestOverReplicatedBlocks and TestWriteReplica failing Sub-task Resolved Jitendra Nath Pandey
          57.
          		 Federation: Bump up LAYOUT_VERSION in FSConstants.java for Federation before committing to trunk Sub-task Resolved Unassigned
          58.
          		 HDFS federation :TestHeartbeatHandling fails: wrong number of items in cmds array Sub-task Resolved Tanping Wang
          59.
          		 HDFS federation : Fix TestBackupNode and TestRefreshNamenodes test failure. Sub-task Resolved Tanping Wang
          60.
          		 HDFS federation: Improve start/stop scripts and add script to decommission datanodes Sub-task Closed Tanping Wang
          61.
          		 Federation: Add a tool that lists namenodes, secondary and backup nodes from the configuration file Sub-task Resolved Suresh Srinivas
          62.
          		 HDFS federation: Balancer command throws NullPointerException Sub-task Resolved Suresh Srinivas
          63.
          		 Hdfs Federation: TestFileAppend2, TestFileAppend3 and TestBlockTokenWithDFS failing Sub-task Resolved Jitendra Nath Pandey
          64.
          		 Hdfs Federation: Failure in browsing data on new namenodes Sub-task Resolved Jitendra Nath Pandey
          65.
          		 Federation: remove datanode's datanodeId TODOs Sub-task Resolved Boris Shkolnik
          66.
          		 Error "nnaddr url param is null" when clicking on a node from NN Live Node Link Sub-task Resolved Jitendra Nath Pandey
          67.
          		 HDFS Federation: create method for updating machine name in DataNode.java Sub-task Resolved Boris Shkolnik
          68.
          		 HDFS Federation: when looking up datanode we should use machineNmae (in testOverReplicatedBlocks) Sub-task Resolved Boris Shkolnik
          69.
          		 Hdfs Federation: Prevent DataBlockScanner from running in tight loop Sub-task Resolved Jitendra Nath Pandey
          70.
          		 HDFS Federation: refactor upgrade object in DataNode Sub-task Resolved Boris Shkolnik
          71.
          		 HDFS Federation: warning/error not generated when datanode sees inconsistent/different Cluster ID between namenodes Sub-task Resolved Boris Shkolnik
          72.
          		 Federation: Add decommission tests for federated namenodes Sub-task Resolved Suresh Srinivas
          73.
          		 Federation: FSDataset volumeMap access is not correctly synchronized Sub-task Resolved Suresh Srinivas
          74.
          		 HDFS Federation: MiniDFSCluster#waitActive() bug causes some tests to fail Sub-task Resolved Suresh Srinivas
          75.
          		 Federation: TestDFSRemove fails intermittently Sub-task Resolved Suresh Srinivas
          76.
          		 Federation: FSVolumeSet volumes is not synchronized correctly Sub-task Resolved Suresh Srinivas
          77.
          		 Hdfs Federation: Configuration for principal names should not be namenode specific. Sub-task Resolved Jitendra Nath Pandey
          78.
          		 HDFS Federation: Add flag to MiniDFSCluster to differentiate between two different modes-Federation and not. Sub-task Resolved Boris Shkolnik
          79.
          		 Federation: merge FSImage changes into FSImage+NNStorage refactoring in trunk Sub-task Resolved Suresh Srinivas
          80.
          		 Federation: Update the layout version for federation changes Sub-task Resolved Suresh Srinivas
          81.
          		 Federation: Add new layout version to offline image/edits viewer Sub-task Resolved Suresh Srinivas
          82.
          		 Federation: fix fault injection test build failure Sub-task Resolved Suresh Srinivas
          83.
          		 Hdfs Federation: TestFileAppend3 fails intermittently. Sub-task Resolved Jitendra Nath Pandey
          84.
          		 HDFS Federation: TestBackupNode Fails on Federation branch Sub-task Resolved Boris Shkolnik
          85.
          		 Hdfs Federation: TestListCorruptFileBlocks failing in federation branch. Sub-task Resolved Jitendra Nath Pandey
          86.
          		 Federation HDFS: testFsck fails Sub-task Resolved Boris Shkolnik
          87.
          		 Hdfs Federation: The BPOfferService must always connect to namenode as the login user. Sub-task Resolved Jitendra Nath Pandey
          88.
          		 Hdfs Federation: Add command to delete block pool directories from a datanode. Sub-task Resolved Jitendra Nath Pandey
          89.
          		 Federation: DatablockScanner should scan blocks for all the block pools. Sub-task Resolved Jitendra Nath Pandey
          90.
          		 Address all the federation TODOs Sub-task Open Suresh Srinivas
          91.
          		 Enable a single 2nn to checkpoint multiple nameservices Sub-task Open Unassigned

          Activity

            People

              sureshms Suresh Srinivas
              sureshms Suresh Srinivas
              Votes:
              1 Vote for this issue
              Watchers:
              76 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: