Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2832

Enable support for heterogeneous storages in HDFS - DN as a collection of storages

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.3.0
    • Component/s: datanode, namenode
    • Labels:
      None
    • Target Version/s:

      Description

      HDFS currently supports configuration where storages are a list of directories. Typically each of these directories correspond to a volume with its own file system. All these directories are homogeneous and therefore identified as a single storage at the namenode. I propose, change to the current model where Datanode * is a * storage, to Datanode * is a collection * of strorages.

        Attachments

        1. 20130813-HeterogeneousStorage.pdf
          80 kB
          Arpit Agarwal
        2. h2832_20131023.patch
          455 kB
          Tsz-wo Sze
        3. h2832_20131023b.patch
          452 kB
          Tsz-wo Sze
        4. h2832_20131025.patch
          456 kB
          Tsz-wo Sze
        5. h2832_20131028.patch
          506 kB
          Arpit Agarwal
        6. h2832_20131028b.patch
          506 kB
          Junping Du
        7. h2832_20131029.patch
          505 kB
          Arpit Agarwal
        8. h2832_20131103.patch
          497 kB
          Tsz-wo Sze
        9. h2832_20131104.patch
          500 kB
          Tsz-wo Sze
        10. h2832_20131105.patch
          517 kB
          Arpit Agarwal
        11. H2832_20131107.patch
          518 kB
          Arpit Agarwal
        12. h2832_20131107b.patch
          519 kB
          Arpit Agarwal
        13. h2832_20131108.patch
          505 kB
          Tsz-wo Sze
        14. h2832_20131110.patch
          541 kB
          Arpit Agarwal
        15. h2832_20131110b.patch
          564 kB
          Arpit Agarwal
        16. h2832_20131111.patch
          563 kB
          Arpit Agarwal
        17. h2832_20131112.patch
          563 kB
          Arpit Agarwal
        18. h2832_20131112b.patch
          564 kB
          Arpit Agarwal
        19. h2832_20131114.patch
          567 kB
          Arpit Agarwal
        20. h2832_20131118.patch
          569 kB
          Arpit Agarwal
        21. h2832_20131119.patch
          568 kB
          Arpit Agarwal
        22. h2832_20131119b.patch
          565 kB
          Arpit Agarwal
        23. h2832_20131121.patch
          564 kB
          Arpit Agarwal
        24. h2832_20131122.patch
          575 kB
          Arpit Agarwal
        25. h2832_20131122b.patch
          562 kB
          Tsz-wo Sze
        26. h2832_20131123.patch
          568 kB
          Tsz-wo Sze
        27. h2832_20131124.patch
          576 kB
          Arpit Agarwal
        28. 20131125-HeterogeneousStorage.pdf
          80 kB
          Arpit Agarwal
        29. 20131125-HeterogeneousStorage-TestPlan.pdf
          44 kB
          Arpit Agarwal
        30. editsStored
          5 kB
          Arpit Agarwal
        31. h2832_20131202.patch
          576 kB
          Arpit Agarwal
        32. 20131202-HeterogeneousStorage-TestPlan.pdf
          44 kB
          Arpit Agarwal
        33. 20131203-HeterogeneousStorage-TestPlan.pdf
          44 kB
          Arpit Agarwal
        34. h2832_20131203.patch
          577 kB
          Arpit Agarwal
        35. h2832_20131210.patch
          578 kB
          Arpit Agarwal
        36. h2832_20131211.patch
          574 kB
          Arpit Agarwal
        37. h2832_20131211b.patch
          574 kB
          Arpit Agarwal
        38. h2832_branch-2_20131226.patch
          566 kB
          Arpit Agarwal
        39. editsStored
          4 kB
          Arpit Agarwal
        40. h2832_branch-2_20140103.patch
          566 kB
          Arpit Agarwal

        Issue Links

        1.
        NamenodeProtocol#getBlocks() should use DatanodeID as an argument instead of DatanodeInfo Sub-task Patch Available Suresh Srinivas Actions
        2.
        Rename protobuf message StorageInfoProto to NodeInfoProto Sub-task Patch Available Suresh Srinivas Actions
        3.
        Protocol buffer changes to add support multiple storages Sub-task Closed Suresh Srinivas Actions
        4.
        Service protocol change to support multiple storages added in HDFS-2880 Sub-task Closed Suresh Srinivas Actions
        5.
        Add apache license header for StorageReport.java Sub-task Closed Brandon Li Actions
        6.
        Change Datanode not to send storage list in registration - it will be sent in block report Sub-task Closed Tsz-wo Sze Actions
        7.
        Add DatanodeStorage information to block recovery Sub-task Closed Tsz-wo Sze Actions
        8.
        Add storage type to the protocol and expose it in block report and block locations Sub-task Resolved Arpit Agarwal Actions
        9.
        Make StorageID a UUID Sub-task Resolved Arpit Agarwal Actions
        10.
        DataNode configuration should allow specifying storage type Sub-task Resolved Arpit Agarwal Actions
        11.
        Namenode changes to track multiple storages Sub-task Resolved Tsz-wo Sze Actions
        12.
        NameNode should expose storage information in the LocatedBlock Sub-task Resolved Tsz-wo Sze Actions
        13.
        NameNode improvements to per-storage block processing Sub-task Resolved Tsz-wo Sze Actions
        14.
        Add StorageType to FsVolume Sub-task Resolved Junping Du Actions
        15.
        Merge HDFS-5077 fix into branch HDFS-2832 Sub-task Resolved Arpit Agarwal Actions
        16.
        Fix several tests failure after HDFS-4987 Sub-task Resolved Junping Du Actions
        17.
        Block placement for storage types Sub-task Resolved Tsz-wo Sze Actions
        18.
        Protocol changes to transmit StorageUuid Sub-task Resolved Arpit Agarwal Actions
        19.
        Deprecate API of getStorageID() and setStorageID() in DatanodeID as multiple storages Sub-task Resolved Junping Du Actions
        20.
        Use Datanode UUID to identify Datanodes Sub-task Resolved Arpit Agarwal Actions
        21.
        Move block schedule information from DatanodeDescriptor to DatanodeStorageInfo Sub-task Resolved Tsz-wo Sze Actions
        22.
        Datanode must support all the volumes as individual storages Sub-task Resolved Arpit Agarwal Actions
        23.
        Heartbeats from Datandode should include one storage report per storage directory Sub-task Resolved Arpit Agarwal Actions
        24.
        NameNode changes to process storage reports per storage directory Sub-task Resolved Arpit Agarwal Actions
        25.
        Send one incremental block report per storage directory Sub-task Resolved Arpit Agarwal Actions
        26.
        Fix NPE in Directory Scanner for HDFS-2832 Sub-task Resolved Arpit Agarwal Actions
        27.
        Fix NPEs in BlockManager and DirectoryScanner Sub-task Resolved Arpit Agarwal Actions
        28.
        Fix storage IDs in PBHelper and UpgradeUtilities Sub-task Resolved Tsz-wo Sze Actions
        29.
        Verify initializations of LocatedBlock/RecoveringBlock Sub-task Resolved Arpit Agarwal Actions
        30.
        Fix storage IDs in PBHelper and UpgradeUtilities Sub-task Resolved Tsz-wo Sze Actions
        31.
        Fix NPE in PBHelper Sub-task Resolved Junping Du Actions
        32.
        File append fails to initialize storageIDs Sub-task Resolved Junping Du Actions
        33.
        Fix TestBlockReport and TestBPOfferService failures Sub-task Resolved Arpit Agarwal Actions
        34.
        Fix TestJspHelper for HDFS-2832 Sub-task Resolved Arpit Agarwal Actions
        35.
        Fix TestReplicationPolicy and TestBlocksScheduledCounter Sub-task Resolved Tsz-wo Sze Actions
        36.
        Datanode should generate its ID on first registration Sub-task Resolved Arpit Agarwal Actions
        37.
        NN should update storageMap on first heartbeat Sub-task Resolved Arpit Agarwal Actions
        38.
        Fix TestDatanodeRegistration, TestFsck and TestAddBlockRetry Sub-task Resolved Tsz-wo Sze Actions
        39.
        Update storage IDs when the pipeline is updated Sub-task Resolved Tsz-wo Sze Actions
        40.
        Fix TestPendingReplication for HDFS-2832 Sub-task Resolved Arpit Agarwal Actions
        41.
        Add back trunk's reportDiff algorithm to HDFS-2832 Sub-task Resolved Tsz-wo Sze Actions
        42.
        Fix TestDatanodeManager, TestSafeMode and TestNNThroughputBenchmark Sub-task Resolved Tsz-wo Sze Actions
        43.
        NN incorrectly tracks more than one replica per DN Sub-task Resolved Arpit Agarwal Actions
        44.
        Fix test failures in Balancer. Sub-task Resolved Unassigned Actions
        45.
        Fix TestDataNodeVolumeFailure Sub-task Resolved Junping Du Actions
        46.
        Update Balancer for HDFS-2832 Sub-task Resolved Tsz-wo Sze Actions
        47.
        Fix TestNameNodeMetrics for HDFS-2832 Sub-task Resolved Arpit Agarwal Actions
        48.
        Update stored edits for HDFS-2832 Sub-task Resolved Arpit Agarwal Actions
        49.
        Fix findbugs warnings for HDFS-2832 Sub-task Resolved Arpit Agarwal Actions
        50.
        Fix pendingReceivedRequests tracking in BPServiceActor Sub-task Resolved Arpit Agarwal Actions
        51.
        Fix compilation error after merge Sub-task Resolved Tsz-wo Sze Actions
        52.
        Fix a findbug warning in DataStorage.java on HDFS-2832 branch Sub-task Resolved Junping Du Actions
        53.
        Fix TestDFSStartupVersions for HDFS-2832 Sub-task Resolved Arpit Agarwal Actions
        54.
        Fix TestUnderReplicatedBlocks on branch HDFS-2832 Sub-task Resolved Arpit Agarwal Actions
        55.
        StorageType and State in DatanodeStorageInfo in NameNode is not accurate Sub-task Resolved Eric Sirianni Actions
        56.
        Fix build break after merge from trunk to HDFS-2832 Sub-task Resolved Arpit Agarwal Actions
        57.
        Fix TestDatanodeConfig in HDFS-2832 Sub-task Resolved Tsz-wo Sze Actions
        58.
        Fix TODO and clean up the code in HDFS-2832. Sub-task Resolved Tsz-wo Sze Actions
        59.
        Remove FsDatasetSpi#getBlockReport Sub-task Resolved Arpit Agarwal Actions
        60.
        Get rid of FsDatasetImpl#perVolumeReplicaMap Sub-task Resolved Arpit Agarwal Actions
        61.
        DataNode UUID should be assigned prior to FsDataset initialization Sub-task Closed Arpit Agarwal Actions
        62.
        Test cases for split and combined block reports Sub-task Resolved Arpit Agarwal Actions
        63.
        Send incremental block reports for all storages in a single call Sub-task Closed Arpit Agarwal Actions
        64.
        Include DatanodeStorage in StorageReport Sub-task Closed Arpit Agarwal Actions
        65.
        Datanode should have compatibility mode for sending combined block reports Sub-task Resolved Arpit Agarwal Actions
        66.
        NN should gracefully handle multiple block replicas on same DN Sub-task Closed Arpit Agarwal Actions

          Activity

            People

            • Assignee:
              arp Arpit Agarwal
              Reporter:
              sureshms Suresh Srinivas

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment