Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: balancer & mover, namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In most of the Hadoop clusters, as more and more data is stored for longer time, the demand for storage is outstripping the compute. Hadoop needs a cost effective and easy to manage solution to meet this demand for storage. Current solution is:

      • Delete the old unused data. This comes at operational cost of identifying unnecessary data and deleting them manually.
      • Add more nodes to the clusters. This adds along with storage capacity unnecessary compute capacity to the cluster.

      Hadoop needs a solution to decouple growing storage capacity from compute capacity. Nodes with higher density and less expensive storage with low compute power are becoming available and can be used as cold storage in the clusters. Based on policy the data from hot storage can be moved to cold storage. Adding more nodes to the cold storage can grow the storage independent of the compute capacity in the cluster.

        Attachments

        1. h6584_20140918b.patch
          423 kB
          Tsz Wo Nicholas Sze
        2. h6584_20140918.patch
          425 kB
          Tsz Wo Nicholas Sze
        3. h6584_20140917b.patch
          420 kB
          Jing Zhao
        4. h6584_20140917.patch
          420 kB
          Jing Zhao
        5. h6584_20140916.patch
          400 kB
          Jing Zhao
        6. h6584_20140916.patch
          402 kB
          Jing Zhao
        7. h6584_20140915.patch
          397 kB
          Jing Zhao
        8. h6584_20140911b.patch
          394 kB
          Jing Zhao
        9. h6584_20140911.patch
          395 kB
          Tsz Wo Nicholas Sze
        10. h6584_20140908b.patch
          388 kB
          Jing Zhao
        11. archival-storage-testplan.pdf
          45 kB
          Jing Zhao
        12. h6584_20140908.patch
          373 kB
          Tsz Wo Nicholas Sze
        13. h6584_20140907.patch
          372 kB
          Tsz Wo Nicholas Sze
        14. HDFS-6584.000.patch
          344 kB
          Jing Zhao
        15. HDFSArchivalStorageDesign20140715.pdf
          166 kB
          Tsz Wo Nicholas Sze
        16. HDFSArchivalStorageDesign20140623.pdf
          152 kB
          Tsz Wo Nicholas Sze

          Issue Links

          There are no Sub-Tasks for this issue.

            Activity

              People

              • Assignee:
                szetszwo Tsz Wo Nicholas Sze
                Reporter:
                szetszwo Tsz Wo Nicholas Sze
              • Votes:
                0 Vote for this issue
                Watchers:
                47 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: