Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2832

Enable support for heterogeneous storages in HDFS - DN as a collection of storages

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.3.0
    • Component/s: datanode, namenode
    • Labels:
      None
    • Target Version/s:

      Description

      HDFS currently supports configuration where storages are a list of directories. Typically each of these directories correspond to a volume with its own file system. All these directories are homogeneous and therefore identified as a single storage at the namenode. I propose, change to the current model where Datanode * is a * storage, to Datanode * is a collection * of strorages.

        Attachments

        1. 20130813-HeterogeneousStorage.pdf
          80 kB
          Arpit Agarwal
        2. 20131125-HeterogeneousStorage.pdf
          80 kB
          Arpit Agarwal
        3. 20131125-HeterogeneousStorage-TestPlan.pdf
          44 kB
          Arpit Agarwal
        4. 20131202-HeterogeneousStorage-TestPlan.pdf
          44 kB
          Arpit Agarwal
        5. 20131203-HeterogeneousStorage-TestPlan.pdf
          44 kB
          Arpit Agarwal
        6. editsStored
          4 kB
          Arpit Agarwal
        7. editsStored
          5 kB
          Arpit Agarwal
        8. h2832_20131023.patch
          455 kB
          Tsz Wo Nicholas Sze
        9. h2832_20131023b.patch
          452 kB
          Tsz Wo Nicholas Sze
        10. h2832_20131025.patch
          456 kB
          Tsz Wo Nicholas Sze
        11. h2832_20131028.patch
          506 kB
          Arpit Agarwal
        12. h2832_20131028b.patch
          506 kB
          Junping Du
        13. h2832_20131029.patch
          505 kB
          Arpit Agarwal
        14. h2832_20131103.patch
          497 kB
          Tsz Wo Nicholas Sze
        15. h2832_20131104.patch
          500 kB
          Tsz Wo Nicholas Sze
        16. h2832_20131105.patch
          517 kB
          Arpit Agarwal
        17. H2832_20131107.patch
          518 kB
          Arpit Agarwal
        18. h2832_20131107b.patch
          519 kB
          Arpit Agarwal
        19. h2832_20131108.patch
          505 kB
          Tsz Wo Nicholas Sze
        20. h2832_20131110.patch
          541 kB
          Arpit Agarwal
        21. h2832_20131110b.patch
          564 kB
          Arpit Agarwal
        22. h2832_20131111.patch
          563 kB
          Arpit Agarwal
        23. h2832_20131112.patch
          563 kB
          Arpit Agarwal
        24. h2832_20131112b.patch
          564 kB
          Arpit Agarwal
        25. h2832_20131114.patch
          567 kB
          Arpit Agarwal
        26. h2832_20131118.patch
          569 kB
          Arpit Agarwal
        27. h2832_20131119.patch
          568 kB
          Arpit Agarwal
        28. h2832_20131119b.patch
          565 kB
          Arpit Agarwal
        29. h2832_20131121.patch
          564 kB
          Arpit Agarwal
        30. h2832_20131122.patch
          575 kB
          Arpit Agarwal
        31. h2832_20131122b.patch
          562 kB
          Tsz Wo Nicholas Sze
        32. h2832_20131123.patch
          568 kB
          Tsz Wo Nicholas Sze
        33. h2832_20131124.patch
          576 kB
          Arpit Agarwal
        34. h2832_20131202.patch
          576 kB
          Arpit Agarwal
        35. h2832_20131203.patch
          577 kB
          Arpit Agarwal
        36. h2832_20131210.patch
          578 kB
          Arpit Agarwal
        37. h2832_20131211.patch
          574 kB
          Arpit Agarwal
        38. h2832_20131211b.patch
          574 kB
          Arpit Agarwal
        39. h2832_branch-2_20131226.patch
          566 kB
          Arpit Agarwal
        40. h2832_branch-2_20140103.patch
          566 kB
          Arpit Agarwal

          Issue Links

          1.
          NamenodeProtocol#getBlocks() should use DatanodeID as an argument instead of DatanodeInfo Sub-task Patch Available Suresh Srinivas
          2.
          Rename protobuf message StorageInfoProto to NodeInfoProto Sub-task Patch Available Suresh Srinivas
          3.
          Protocol buffer changes to add support multiple storages Sub-task Closed Suresh Srinivas
          4.
          Service protocol change to support multiple storages added in HDFS-2880 Sub-task Closed Suresh Srinivas
          5.
          Add apache license header for StorageReport.java Sub-task Closed Brandon Li
          6.
          Change Datanode not to send storage list in registration - it will be sent in block report Sub-task Closed Tsz Wo Nicholas Sze
          7.
          Add DatanodeStorage information to block recovery Sub-task Closed Tsz Wo Nicholas Sze
          8.
          Add storage type to the protocol and expose it in block report and block locations Sub-task Resolved Arpit Agarwal
          9.
          Make StorageID a UUID Sub-task Resolved Arpit Agarwal
          10.
          DataNode configuration should allow specifying storage type Sub-task Resolved Arpit Agarwal
          11.
          Namenode changes to track multiple storages Sub-task Resolved Tsz Wo Nicholas Sze
          12.
          NameNode should expose storage information in the LocatedBlock Sub-task Resolved Tsz Wo Nicholas Sze
          13.
          NameNode improvements to per-storage block processing Sub-task Resolved Tsz Wo Nicholas Sze
          14.
          Add StorageType to FsVolume Sub-task Resolved Junping Du
          15.
          Merge HDFS-5077 fix into branch HDFS-2832 Sub-task Resolved Arpit Agarwal
          16.
          Fix several tests failure after HDFS-4987 Sub-task Resolved Junping Du
          17.
          Block placement for storage types Sub-task Resolved Tsz Wo Nicholas Sze
          18.
          Protocol changes to transmit StorageUuid Sub-task Resolved Arpit Agarwal
          19.
          Deprecate API of getStorageID() and setStorageID() in DatanodeID as multiple storages Sub-task Resolved Junping Du
          20.
          Use Datanode UUID to identify Datanodes Sub-task Resolved Arpit Agarwal
          21.
          Move block schedule information from DatanodeDescriptor to DatanodeStorageInfo Sub-task Resolved Tsz Wo Nicholas Sze
          22.
          Datanode must support all the volumes as individual storages Sub-task Resolved Arpit Agarwal
          23.
          Heartbeats from Datandode should include one storage report per storage directory Sub-task Resolved Arpit Agarwal
          24.
          NameNode changes to process storage reports per storage directory Sub-task Resolved Arpit Agarwal
          25.
          Send one incremental block report per storage directory Sub-task Resolved Arpit Agarwal
          26.
          Fix NPE in Directory Scanner for HDFS-2832 Sub-task Resolved Arpit Agarwal
          27.
          Fix NPEs in BlockManager and DirectoryScanner Sub-task Resolved Arpit Agarwal
          28.
          Fix storage IDs in PBHelper and UpgradeUtilities Sub-task Resolved Tsz Wo Nicholas Sze
          29.
          Verify initializations of LocatedBlock/RecoveringBlock Sub-task Resolved Arpit Agarwal
          30.
          Fix storage IDs in PBHelper and UpgradeUtilities Sub-task Resolved Tsz Wo Nicholas Sze
          31.
          Fix NPE in PBHelper Sub-task Resolved Junping Du
          32.
          File append fails to initialize storageIDs Sub-task Resolved Junping Du
          33.
          Fix TestBlockReport and TestBPOfferService failures Sub-task Resolved Arpit Agarwal
          34.
          Fix TestJspHelper for HDFS-2832 Sub-task Resolved Arpit Agarwal
          35.
          Fix TestReplicationPolicy and TestBlocksScheduledCounter Sub-task Resolved Tsz Wo Nicholas Sze
          36.
          Datanode should generate its ID on first registration Sub-task Resolved Arpit Agarwal
          37.
          NN should update storageMap on first heartbeat Sub-task Resolved Arpit Agarwal
          38.
          Fix TestDatanodeRegistration, TestFsck and TestAddBlockRetry Sub-task Resolved Tsz Wo Nicholas Sze
          39.
          Update storage IDs when the pipeline is updated Sub-task Resolved Tsz Wo Nicholas Sze
          40.
          Fix TestPendingReplication for HDFS-2832 Sub-task Resolved Arpit Agarwal
          41.
          Add back trunk's reportDiff algorithm to HDFS-2832 Sub-task Resolved Tsz Wo Nicholas Sze
          42.
          Fix TestDatanodeManager, TestSafeMode and TestNNThroughputBenchmark Sub-task Resolved Tsz Wo Nicholas Sze
          43.
          NN incorrectly tracks more than one replica per DN Sub-task Resolved Arpit Agarwal
          44.
          Fix test failures in Balancer. Sub-task Resolved Unassigned
          45.
          Fix TestDataNodeVolumeFailure Sub-task Resolved Junping Du
          46.
          Update Balancer for HDFS-2832 Sub-task Resolved Tsz Wo Nicholas Sze
          47.
          Fix TestNameNodeMetrics for HDFS-2832 Sub-task Resolved Arpit Agarwal
          48.
          Update stored edits for HDFS-2832 Sub-task Resolved Arpit Agarwal
          49.
          Fix findbugs warnings for HDFS-2832 Sub-task Resolved Arpit Agarwal
          50.
          Fix pendingReceivedRequests tracking in BPServiceActor Sub-task Resolved Arpit Agarwal
          51.
          Fix compilation error after merge Sub-task Resolved Tsz Wo Nicholas Sze
          52.
          Fix a findbug warning in DataStorage.java on HDFS-2832 branch Sub-task Resolved Junping Du
          53.
          Fix TestDFSStartupVersions for HDFS-2832 Sub-task Resolved Arpit Agarwal
          54.
          Fix TestUnderReplicatedBlocks on branch HDFS-2832 Sub-task Resolved Arpit Agarwal
          55.
          StorageType and State in DatanodeStorageInfo in NameNode is not accurate Sub-task Resolved Eric Sirianni
          56.
          Fix build break after merge from trunk to HDFS-2832 Sub-task Resolved Arpit Agarwal
          57.
          Fix TestDatanodeConfig in HDFS-2832 Sub-task Resolved Tsz Wo Nicholas Sze
          58.
          Fix TODO and clean up the code in HDFS-2832. Sub-task Resolved Tsz Wo Nicholas Sze
          59.
          Remove FsDatasetSpi#getBlockReport Sub-task Resolved Arpit Agarwal
          60.
          Get rid of FsDatasetImpl#perVolumeReplicaMap Sub-task Resolved Arpit Agarwal
          61.
          DataNode UUID should be assigned prior to FsDataset initialization Sub-task Closed Arpit Agarwal
          62.
          Test cases for split and combined block reports Sub-task Resolved Arpit Agarwal
          63.
          Send incremental block reports for all storages in a single call Sub-task Closed Arpit Agarwal
          64.
          Include DatanodeStorage in StorageReport Sub-task Closed Arpit Agarwal
          65.
          Datanode should have compatibility mode for sending combined block reports Sub-task Resolved Arpit Agarwal
          66.
          NN should gracefully handle multiple block replicas on same DN Sub-task Closed Arpit Agarwal

            Activity

              People

              • Assignee:
                arpitagarwal Arpit Agarwal
                Reporter:
                sureshms Suresh Srinivas
              • Votes:
                0 Vote for this issue
                Watchers:
                91 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: