Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9725

Make capacity of centralized cache dynamic

    XMLWordPrintableJSON

    Details

    • Type: Wish
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.6.0
    • Fix Version/s: None
    • Component/s: caching, datanode, namenode
    • Labels:
      None

      Description

      Currently in centralized cache, Datanode uses mlock to keep blocks in memory, with limit of maximum amount of bytes specified by dnConf.maxLockedMemory.
      In general deployment, each machine run both Datanode and Nodemanager. In this case, statically specified memory capacity either potentially causes OOM, or hurts memory utilization.
      That is, if one specify a large capacity for caching (permitted by ulimit as prerequisite), Datanode may have gone too far to reserve any memory for new container process to launch from NodeManager. On the other hand, specifying a small value may leave memory insufficiently used.

      A simple idea is: perhaps it is better to make cache capacity dynamic. Adjusting its capacity corresponding to current (or future, ideally) memory usage to avoid problems above.

      Any suggestions or comments?

        Attachments

          Activity

            People

            • Assignee:
              He Tianyi He Tianyi
              Reporter:
              He Tianyi He Tianyi
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: