Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4630

Datanode is going OOM due to small files in hdfs

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Invalid
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: None
    • Component/s: datanode, namenode
    • Labels:
      None
    • Environment:

      Ubuntu, Java 1.6

      Description

      Hi,

      We have very small files(size ranging 10KB-1MB) in our hdfs and no of files are in tens of millions. Due to this namenode and datanode both going out of memory very frequently. When we analyse the head dump of datanode most of the memory was used by ReplicaMap.

      Can we use EhCache or other to not to store all the data in memory?

      Thanks
      Ankush

        Activity

        Hide
        Steve Loughran added a comment -

        I'd say "WONTFIX" over invalid; the OOM is a result of storing all state in memory for bounded time operations against files, including block retrieval. That's a design decision. Now, if you want to put EhCache in behind the scenes, assess its performance with many small files, and its behaviour on big production clusters, that's a project I'm sure we'd all be curious about -feel free to have a go!

        Show
        Steve Loughran added a comment - I'd say "WONTFIX" over invalid; the OOM is a result of storing all state in memory for bounded time operations against files, including block retrieval. That's a design decision. Now, if you want to put EhCache in behind the scenes, assess its performance with many small files, and its behaviour on big production clusters, that's a project I'm sure we'd all be curious about -feel free to have a go!
        Hide
        Harsh J added a comment -

        Closing again per Suresh's comment, as this is by design and you're merely required to raise your heap to accommodate more files (and thereby, blocks). Please also see HDFS-4465 and HDFS-4461 on optimizations of this.

        Show
        Harsh J added a comment - Closing again per Suresh's comment, as this is by design and you're merely required to raise your heap to accommodate more files (and thereby, blocks). Please also see HDFS-4465 and HDFS-4461 on optimizations of this.
        Hide
        Suresh Srinivas added a comment -

        this is not question, Its an issue Datanode going OOM due to incapability of holding all the pointers to all the blocks.

        The datanode JVM needs to be sized correctly for holding the blocks. Datanodes consume memory proportional to the number of block replicas on a datanode. Hence I said, please get information on how to size the process.

        Show
        Suresh Srinivas added a comment - this is not question, Its an issue Datanode going OOM due to incapability of holding all the pointers to all the blocks. The datanode JVM needs to be sized correctly for holding the blocks. Datanodes consume memory proportional to the number of block replicas on a datanode. Hence I said, please get information on how to size the process.
        Hide
        Ankush Bhatiya added a comment -

        this is not question, Its an issue Datanode going OOM due to incapability of holding all the pointers to all the blocks.

        Show
        Ankush Bhatiya added a comment - this is not question, Its an issue Datanode going OOM due to incapability of holding all the pointers to all the blocks.
        Hide
        Suresh Srinivas added a comment -

        Jiras are for reporting bugs and not a forum for asking questions. Please discuss this in user mailing list on how to size the namenode and datanode processes.

        Show
        Suresh Srinivas added a comment - Jiras are for reporting bugs and not a forum for asking questions. Please discuss this in user mailing list on how to size the namenode and datanode processes.

          People

          • Assignee:
            Unassigned
            Reporter:
            Ankush Bhatiya
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development