Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4949

Centralized cache management in HDFS

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.3.0, 3.0.0-alpha1
    • 2.3.0
    • datanode, namenode
    • None

    Description

      HDFS currently has no support for managing or exposing in-memory caches at datanodes. This makes it harder for higher level application frameworks like Hive, Pig, and Impala to effectively use cluster memory, because they cannot explicitly cache important datasets or place their tasks for memory locality.

      Attachments

        1. caching-design-doc-2013-07-02.pdf
          270 kB
          Andrew Wang
        2. caching-design-doc-2013-08-09.pdf
          305 kB
          Andrew Wang
        3. caching-design-doc-2013-10-24.pdf
          312 kB
          Colin McCabe
        4. caching-testplan.pdf
          99 kB
          Stephen Chu
        5. hdfs-4949-branch-2.patch
          698 kB
          Andrew Wang
        6. HDFS-4949-consolidated.patch
          503 kB
          Andrew Wang

        Issue Links

          There are no Sub-Tasks for this issue.

          Activity

            People

              andrew.wang Andrew Wang
              andrew.wang Andrew Wang
              Votes:
              0 Vote for this issue
              Watchers:
              106 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: