Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2832

Enable support for heterogeneous storages in HDFS - DN as a collection of storages

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.3.0
    • Component/s: datanode, namenode
    • Labels:
      None
    • Target Version/s:

      Description

      HDFS currently supports configuration where storages are a list of directories. Typically each of these directories correspond to a volume with its own file system. All these directories are homogeneous and therefore identified as a single storage at the namenode. I propose, change to the current model where Datanode * is a * storage, to Datanode * is a collection * of strorages.

      1. 20130813-HeterogeneousStorage.pdf
        80 kB
        Arpit Agarwal
      2. h2832_20131023.patch
        455 kB
        Tsz Wo Nicholas Sze
      3. h2832_20131023b.patch
        452 kB
        Tsz Wo Nicholas Sze
      4. h2832_20131025.patch
        456 kB
        Tsz Wo Nicholas Sze
      5. h2832_20131028.patch
        506 kB
        Arpit Agarwal
      6. h2832_20131028b.patch
        506 kB
        Junping Du
      7. h2832_20131029.patch
        505 kB
        Arpit Agarwal
      8. h2832_20131103.patch
        497 kB
        Tsz Wo Nicholas Sze
      9. h2832_20131104.patch
        500 kB
        Tsz Wo Nicholas Sze
      10. h2832_20131105.patch
        517 kB
        Arpit Agarwal
      11. H2832_20131107.patch
        518 kB
        Arpit Agarwal
      12. h2832_20131107b.patch
        519 kB
        Arpit Agarwal
      13. h2832_20131108.patch
        505 kB
        Tsz Wo Nicholas Sze
      14. h2832_20131110.patch
        541 kB
        Arpit Agarwal
      15. h2832_20131110b.patch
        564 kB
        Arpit Agarwal
      16. h2832_20131111.patch
        563 kB
        Arpit Agarwal
      17. h2832_20131112.patch
        563 kB
        Arpit Agarwal
      18. h2832_20131112b.patch
        564 kB
        Arpit Agarwal
      19. h2832_20131114.patch
        567 kB
        Arpit Agarwal
      20. h2832_20131118.patch
        569 kB
        Arpit Agarwal
      21. h2832_20131119.patch
        568 kB
        Arpit Agarwal
      22. h2832_20131119b.patch
        565 kB
        Arpit Agarwal
      23. h2832_20131121.patch
        564 kB
        Arpit Agarwal
      24. h2832_20131122.patch
        575 kB
        Arpit Agarwal
      25. h2832_20131122b.patch
        562 kB
        Tsz Wo Nicholas Sze
      26. h2832_20131123.patch
        568 kB
        Tsz Wo Nicholas Sze
      27. h2832_20131124.patch
        576 kB
        Arpit Agarwal
      28. 20131125-HeterogeneousStorage.pdf
        80 kB
        Arpit Agarwal
      29. 20131125-HeterogeneousStorage-TestPlan.pdf
        44 kB
        Arpit Agarwal
      30. editsStored
        5 kB
        Arpit Agarwal
      31. h2832_20131202.patch
        576 kB
        Arpit Agarwal
      32. 20131202-HeterogeneousStorage-TestPlan.pdf
        44 kB
        Arpit Agarwal
      33. 20131203-HeterogeneousStorage-TestPlan.pdf
        44 kB
        Arpit Agarwal
      34. h2832_20131203.patch
        577 kB
        Arpit Agarwal
      35. h2832_20131210.patch
        578 kB
        Arpit Agarwal
      36. h2832_20131211.patch
        574 kB
        Arpit Agarwal
      37. h2832_20131211b.patch
        574 kB
        Arpit Agarwal
      38. h2832_branch-2_20131226.patch
        566 kB
        Arpit Agarwal
      39. editsStored
        4 kB
        Arpit Agarwal
      40. h2832_branch-2_20140103.patch
        566 kB
        Arpit Agarwal

        Issue Links

        1. NamenodeProtocol#getBlocks() should use DatanodeID as an argument instead of DatanodeInfo Sub-task Patch Available Suresh Srinivas
         
        2. Rename protobuf message StorageInfoProto to NodeInfoProto Sub-task Patch Available Suresh Srinivas
         
        3.
        Protocol buffer changes to add support multiple storages Sub-task Closed Suresh Srinivas
         
        4.
        Service protocol change to support multiple storages added in HDFS-2880 Sub-task Closed Suresh Srinivas
         
        5.
        Add apache license header for StorageReport.java Sub-task Closed Brandon Li
         
        6.
        Change Datanode not to send storage list in registration - it will be sent in block report Sub-task Closed Tsz Wo Nicholas Sze
         
        7.
        Add DatanodeStorage information to block recovery Sub-task Closed Tsz Wo Nicholas Sze
         
        8.
        Add storage type to the protocol and expose it in block report and block locations Sub-task Resolved Arpit Agarwal
         
        9.
        Make StorageID a UUID Sub-task Resolved Arpit Agarwal
         
        10.
        DataNode configuration should allow specifying storage type Sub-task Resolved Arpit Agarwal
         
        11.
        Namenode changes to track multiple storages Sub-task Resolved Tsz Wo Nicholas Sze
         
        12.
        NameNode should expose storage information in the LocatedBlock Sub-task Resolved Tsz Wo Nicholas Sze
         
        13.
        NameNode improvements to per-storage block processing Sub-task Resolved Tsz Wo Nicholas Sze
         
        14.
        Add StorageType to FsVolume Sub-task Resolved Junping Du
         
        15.
        Merge HDFS-5077 fix into branch HDFS-2832 Sub-task Resolved Arpit Agarwal
         
        16.
        Fix several tests failure after HDFS-4987 Sub-task Resolved Junping Du
         
        17.
        Block placement for storage types Sub-task Resolved Tsz Wo Nicholas Sze
         
        18.
        Protocol changes to transmit StorageUuid Sub-task Resolved Arpit Agarwal
         
        19.
        Deprecate API of getStorageID() and setStorageID() in DatanodeID as multiple storages Sub-task Resolved Junping Du
         
        20.
        Use Datanode UUID to identify Datanodes Sub-task Resolved Arpit Agarwal
         
        21.
        Move block schedule information from DatanodeDescriptor to DatanodeStorageInfo Sub-task Resolved Tsz Wo Nicholas Sze
         
        22.
        Datanode must support all the volumes as individual storages Sub-task Resolved Arpit Agarwal
         
        23.
        Heartbeats from Datandode should include one storage report per storage directory Sub-task Resolved Arpit Agarwal
         
        24.
        NameNode changes to process storage reports per storage directory Sub-task Resolved Arpit Agarwal
         
        25.
        Send one incremental block report per storage directory Sub-task Resolved Arpit Agarwal
         
        26.
        Fix NPE in Directory Scanner for HDFS-2832 Sub-task Resolved Arpit Agarwal
         
        27.
        Fix NPEs in BlockManager and DirectoryScanner Sub-task Resolved Arpit Agarwal
         
        28.
        Fix storage IDs in PBHelper and UpgradeUtilities Sub-task Resolved Tsz Wo Nicholas Sze
         
        29.
        Verify initializations of LocatedBlock/RecoveringBlock Sub-task Resolved Arpit Agarwal
         
        30.
        Fix storage IDs in PBHelper and UpgradeUtilities Sub-task Resolved Tsz Wo Nicholas Sze
         
        31.
        Fix NPE in PBHelper Sub-task Resolved Junping Du
         
        32.
        File append fails to initialize storageIDs Sub-task Resolved Junping Du
         
        33.
        Fix TestBlockReport and TestBPOfferService failures Sub-task Resolved Arpit Agarwal
         
        34.
        Fix TestJspHelper for HDFS-2832 Sub-task Resolved Arpit Agarwal
         
        35.
        Fix TestReplicationPolicy and TestBlocksScheduledCounter Sub-task Resolved Tsz Wo Nicholas Sze
         
        36.
        Datanode should generate its ID on first registration Sub-task Resolved Arpit Agarwal
         
        37.
        NN should update storageMap on first heartbeat Sub-task Resolved Arpit Agarwal
         
        38.
        Fix TestDatanodeRegistration, TestFsck and TestAddBlockRetry Sub-task Resolved Tsz Wo Nicholas Sze
         
        39.
        Update storage IDs when the pipeline is updated Sub-task Resolved Tsz Wo Nicholas Sze
         
        40.
        Fix TestPendingReplication for HDFS-2832 Sub-task Resolved Arpit Agarwal
         
        41.
        Add back trunk's reportDiff algorithm to HDFS-2832 Sub-task Resolved Tsz Wo Nicholas Sze
         
        42.
        Fix TestDatanodeManager, TestSafeMode and TestNNThroughputBenchmark Sub-task Resolved Tsz Wo Nicholas Sze
         
        43.
        NN incorrectly tracks more than one replica per DN Sub-task Resolved Arpit Agarwal
         
        44.
        Fix test failures in Balancer. Sub-task Resolved Unassigned
         
        45.
        Fix TestDataNodeVolumeFailure Sub-task Resolved Junping Du
         
        46.
        Update Balancer for HDFS-2832 Sub-task Resolved Tsz Wo Nicholas Sze
         
        47.
        Fix TestNameNodeMetrics for HDFS-2832 Sub-task Resolved Arpit Agarwal
         
        48.
        Update stored edits for HDFS-2832 Sub-task Resolved Arpit Agarwal
         
        49.
        Fix findbugs warnings for HDFS-2832 Sub-task Resolved Arpit Agarwal
         
        50.
        Fix pendingReceivedRequests tracking in BPServiceActor Sub-task Resolved Arpit Agarwal
         
        51.
        Fix compilation error after merge Sub-task Resolved Tsz Wo Nicholas Sze
         
        52.
        Fix a findbug warning in DataStorage.java on HDFS-2832 branch Sub-task Resolved Junping Du
         
        53.
        Fix TestDFSStartupVersions for HDFS-2832 Sub-task Resolved Arpit Agarwal
         
        54.
        Fix TestUnderReplicatedBlocks on branch HDFS-2832 Sub-task Resolved Arpit Agarwal
         
        55.
        StorageType and State in DatanodeStorageInfo in NameNode is not accurate Sub-task Resolved Eric Sirianni
         
        56.
        Fix build break after merge from trunk to HDFS-2832 Sub-task Resolved Arpit Agarwal
         
        57.
        Fix TestDatanodeConfig in HDFS-2832 Sub-task Resolved Tsz Wo Nicholas Sze
         
        58.
        Fix TODO and clean up the code in HDFS-2832. Sub-task Resolved Tsz Wo Nicholas Sze
         
        59.
        Remove FsDatasetSpi#getBlockReport Sub-task Resolved Arpit Agarwal
         
        60.
        Get rid of FsDatasetImpl#perVolumeReplicaMap Sub-task Resolved Arpit Agarwal
         
        61.
        DataNode UUID should be assigned prior to FsDataset initialization Sub-task Closed Arpit Agarwal
         
        62.
        Test cases for split and combined block reports Sub-task Resolved Arpit Agarwal
         
        63.
        Send incremental block reports for all storages in a single call Sub-task Closed Arpit Agarwal
         
        64.
        Include DatanodeStorage in StorageReport Sub-task Closed Arpit Agarwal
         
        65.
        Datanode should have compatibility mode for sending combined block reports Sub-task Resolved Arpit Agarwal
         
        66.
        NN should gracefully handle multiple block replicas on same DN Sub-task Closed Arpit Agarwal
         

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Arpit Agarwal
              Reporter:
              Suresh Srinivas
            • Votes:
              0 Vote for this issue
              Watchers:
              86 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development