Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2832

Enable support for heterogeneous storages in HDFS - DN as a collection of storages

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.0.0-alpha
    • 2.3.0
    • datanode, namenode
    • None

    Description

      HDFS currently supports configuration where storages are a list of directories. Typically each of these directories correspond to a volume with its own file system. All these directories are homogeneous and therefore identified as a single storage at the namenode. I propose, change to the current model where Datanode * is a * storage, to Datanode * is a collection * of strorages.

      Attachments

        1. 20130813-HeterogeneousStorage.pdf
          80 kB
          Arpit Agarwal
        2. h2832_20131023.patch
          455 kB
          Tsz-wo Sze
        3. h2832_20131023b.patch
          452 kB
          Tsz-wo Sze
        4. h2832_20131025.patch
          456 kB
          Tsz-wo Sze
        5. h2832_20131028.patch
          506 kB
          Arpit Agarwal
        6. h2832_20131028b.patch
          506 kB
          Junping Du
        7. h2832_20131029.patch
          505 kB
          Arpit Agarwal
        8. h2832_20131103.patch
          497 kB
          Tsz-wo Sze
        9. h2832_20131104.patch
          500 kB
          Tsz-wo Sze
        10. h2832_20131105.patch
          517 kB
          Arpit Agarwal
        11. H2832_20131107.patch
          518 kB
          Arpit Agarwal
        12. h2832_20131107b.patch
          519 kB
          Arpit Agarwal
        13. h2832_20131108.patch
          505 kB
          Tsz-wo Sze
        14. h2832_20131110.patch
          541 kB
          Arpit Agarwal
        15. h2832_20131110b.patch
          564 kB
          Arpit Agarwal
        16. h2832_20131111.patch
          563 kB
          Arpit Agarwal
        17. h2832_20131112.patch
          563 kB
          Arpit Agarwal
        18. h2832_20131112b.patch
          564 kB
          Arpit Agarwal
        19. h2832_20131114.patch
          567 kB
          Arpit Agarwal
        20. h2832_20131118.patch
          569 kB
          Arpit Agarwal
        21. h2832_20131119.patch
          568 kB
          Arpit Agarwal
        22. h2832_20131119b.patch
          565 kB
          Arpit Agarwal
        23. h2832_20131121.patch
          564 kB
          Arpit Agarwal
        24. h2832_20131122.patch
          575 kB
          Arpit Agarwal
        25. h2832_20131122b.patch
          562 kB
          Tsz-wo Sze
        26. h2832_20131123.patch
          568 kB
          Tsz-wo Sze
        27. h2832_20131124.patch
          576 kB
          Arpit Agarwal
        28. 20131125-HeterogeneousStorage.pdf
          80 kB
          Arpit Agarwal
        29. 20131125-HeterogeneousStorage-TestPlan.pdf
          44 kB
          Arpit Agarwal
        30. editsStored
          5 kB
          Arpit Agarwal
        31. h2832_20131202.patch
          576 kB
          Arpit Agarwal
        32. 20131202-HeterogeneousStorage-TestPlan.pdf
          44 kB
          Arpit Agarwal
        33. 20131203-HeterogeneousStorage-TestPlan.pdf
          44 kB
          Arpit Agarwal
        34. h2832_20131203.patch
          577 kB
          Arpit Agarwal
        35. h2832_20131210.patch
          578 kB
          Arpit Agarwal
        36. h2832_20131211.patch
          574 kB
          Arpit Agarwal
        37. h2832_20131211b.patch
          574 kB
          Arpit Agarwal
        38. h2832_branch-2_20131226.patch
          566 kB
          Arpit Agarwal
        39. editsStored
          4 kB
          Arpit Agarwal
        40. h2832_branch-2_20140103.patch
          566 kB
          Arpit Agarwal

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-6702 DFSClient should create blocks using StorageType

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HDFS-4949 Centralized cache management in HDFS

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HDFS-6584 Support Archival Storage

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-7343 HDFS smart storage management

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-6700 BlockPlacementPolicy shoud choose storage but not datanode for deletion

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          relates to

          Sub-task - The sub-task of the issue HDFS-7390 Provide JMX metrics per storage type

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HDFS-4672 Support tiered storage policies

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HDFS-7285 Erasure Coding Support inside HDFS

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HDFS-5682 Heterogeneous Storage phase 2 - APIs to expose Storage Types

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-3672 Expose disk-location information for blocks to enable better scheduling

          • Major - Major loss of function.
          • Closed

          Brainstorming - A place to record back and forth on notions not yet formed enough to make a 'New Feature' or 'Task' HBASE-6572 Tiered HFile storage

          • Major - Major loss of function.
          • Closed
          1.
          		 NamenodeProtocol#getBlocks() should use DatanodeID as an argument instead of DatanodeInfo Sub-task Patch Available Suresh Srinivas
          2.
          		 Rename protobuf message StorageInfoProto to NodeInfoProto Sub-task Patch Available Suresh Srinivas
          3.
          		 Protocol buffer changes to add support multiple storages Sub-task Closed Suresh Srinivas
          4.
          		 Service protocol change to support multiple storages added in HDFS-2880 Sub-task Closed Suresh Srinivas
          5.
          		 Add apache license header for StorageReport.java Sub-task Closed Brandon Li
          6.
          		 Change Datanode not to send storage list in registration - it will be sent in block report Sub-task Closed Tsz-wo Sze
          7.
          		 Add DatanodeStorage information to block recovery Sub-task Closed Tsz-wo Sze
          8.
          		 Add storage type to the protocol and expose it in block report and block locations Sub-task Resolved Arpit Agarwal
          9.
          		 Make StorageID a UUID Sub-task Resolved Arpit Agarwal
          10.
          		 DataNode configuration should allow specifying storage type Sub-task Resolved Arpit Agarwal
          11.
          		 Namenode changes to track multiple storages Sub-task Resolved Tsz-wo Sze
          12.
          		 NameNode should expose storage information in the LocatedBlock Sub-task Resolved Tsz-wo Sze
          13.
          		 NameNode improvements to per-storage block processing Sub-task Resolved Tsz-wo Sze
          14.
          		 Add StorageType to FsVolume Sub-task Resolved Junping Du
          15.
          		 Merge HDFS-5077 fix into branch HDFS-2832 Sub-task Resolved Arpit Agarwal
          16.
          		 Fix several tests failure after HDFS-4987 Sub-task Resolved Junping Du
          17.
          		 Block placement for storage types Sub-task Resolved Tsz-wo Sze
          18.
          		 Protocol changes to transmit StorageUuid Sub-task Resolved Arpit Agarwal
          19.
          		 Deprecate API of getStorageID() and setStorageID() in DatanodeID as multiple storages Sub-task Resolved Junping Du
          20.
          		 Use Datanode UUID to identify Datanodes Sub-task Resolved Arpit Agarwal
          21.
          		 Move block schedule information from DatanodeDescriptor to DatanodeStorageInfo Sub-task Resolved Tsz-wo Sze
          22.
          		 Datanode must support all the volumes as individual storages Sub-task Resolved Arpit Agarwal
          23.
          		 Heartbeats from Datandode should include one storage report per storage directory Sub-task Resolved Arpit Agarwal
          24.
          		 NameNode changes to process storage reports per storage directory Sub-task Resolved Arpit Agarwal
          25.
          		 Send one incremental block report per storage directory Sub-task Resolved Arpit Agarwal
          26.
          		 Fix NPE in Directory Scanner for HDFS-2832 Sub-task Resolved Arpit Agarwal
          27.
          		 Fix NPEs in BlockManager and DirectoryScanner Sub-task Resolved Arpit Agarwal
          28.
          		 Fix storage IDs in PBHelper and UpgradeUtilities Sub-task Resolved Tsz-wo Sze
          29.
          		 Verify initializations of LocatedBlock/RecoveringBlock Sub-task Resolved Arpit Agarwal
          30.
          		 Fix storage IDs in PBHelper and UpgradeUtilities Sub-task Resolved Tsz-wo Sze
          31.
          		 Fix NPE in PBHelper Sub-task Resolved Junping Du
          32.
          		 File append fails to initialize storageIDs Sub-task Resolved Junping Du
          33.
          		 Fix TestBlockReport and TestBPOfferService failures Sub-task Resolved Arpit Agarwal
          34.
          		 Fix TestJspHelper for HDFS-2832 Sub-task Resolved Arpit Agarwal
          35.
          		 Fix TestReplicationPolicy and TestBlocksScheduledCounter Sub-task Resolved Tsz-wo Sze
          36.
          		 Datanode should generate its ID on first registration Sub-task Resolved Arpit Agarwal
          37.
          		 NN should update storageMap on first heartbeat Sub-task Resolved Arpit Agarwal
          38.
          		 Fix TestDatanodeRegistration, TestFsck and TestAddBlockRetry Sub-task Resolved Tsz-wo Sze
          39.
          		 Update storage IDs when the pipeline is updated Sub-task Resolved Tsz-wo Sze
          40.
          		 Fix TestPendingReplication for HDFS-2832 Sub-task Resolved Arpit Agarwal
          41.
          		 Add back trunk's reportDiff algorithm to HDFS-2832 Sub-task Resolved Tsz-wo Sze
          42.
          		 Fix TestDatanodeManager, TestSafeMode and TestNNThroughputBenchmark Sub-task Resolved Tsz-wo Sze
          43.
          		 NN incorrectly tracks more than one replica per DN Sub-task Resolved Arpit Agarwal
          44.
          		 Fix test failures in Balancer. Sub-task Resolved Unassigned
          45.
          		 Fix TestDataNodeVolumeFailure Sub-task Resolved Junping Du
          46.
          		 Update Balancer for HDFS-2832 Sub-task Resolved Tsz-wo Sze
          47.
          		 Fix TestNameNodeMetrics for HDFS-2832 Sub-task Resolved Arpit Agarwal
          48.
          		 Update stored edits for HDFS-2832 Sub-task Resolved Arpit Agarwal
          49.
          		 Fix findbugs warnings for HDFS-2832 Sub-task Resolved Arpit Agarwal
          50.
          		 Fix pendingReceivedRequests tracking in BPServiceActor Sub-task Resolved Arpit Agarwal
          51.
          		 Fix compilation error after merge Sub-task Resolved Tsz-wo Sze
          52.
          		 Fix a findbug warning in DataStorage.java on HDFS-2832 branch Sub-task Resolved Junping Du
          53.
          		 Fix TestDFSStartupVersions for HDFS-2832 Sub-task Resolved Arpit Agarwal
          54.
          		 Fix TestUnderReplicatedBlocks on branch HDFS-2832 Sub-task Resolved Arpit Agarwal
          55.
          		 StorageType and State in DatanodeStorageInfo in NameNode is not accurate Sub-task Resolved Eric Sirianni
          56.
          		 Fix build break after merge from trunk to HDFS-2832 Sub-task Resolved Arpit Agarwal
          57.
          		 Fix TestDatanodeConfig in HDFS-2832 Sub-task Resolved Tsz-wo Sze
          58.
          		 Fix TODO and clean up the code in HDFS-2832. Sub-task Resolved Tsz-wo Sze
          59.
          		 Remove FsDatasetSpi#getBlockReport Sub-task Resolved Arpit Agarwal
          60.
          		 Get rid of FsDatasetImpl#perVolumeReplicaMap Sub-task Resolved Arpit Agarwal
          61.
          		 DataNode UUID should be assigned prior to FsDataset initialization Sub-task Closed Arpit Agarwal
          62.
          		 Test cases for split and combined block reports Sub-task Resolved Arpit Agarwal
          63.
          		 Send incremental block reports for all storages in a single call Sub-task Closed Arpit Agarwal
          64.
          		 Include DatanodeStorage in StorageReport Sub-task Closed Arpit Agarwal
          65.
          		 Datanode should have compatibility mode for sending combined block reports Sub-task Resolved Arpit Agarwal
          66.
          		 NN should gracefully handle multiple block replicas on same DN Sub-task Closed Arpit Agarwal

          Activity

            People

              arp Arpit Agarwal
              sureshms Suresh Srinivas
              Votes:
              0 Vote for this issue
              Watchers:
              88 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: