Uploaded image for project: 'Hadoop Distributed Data Store'
  1. Hadoop Distributed Data Store
  2. HDDS-2331

Client OOME due to buffer retention

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: None
    • Component/s: Ozone Client
    • Labels:
      None
    • Target Version/s:

      Description

      Freon random key generator exhausts default heap after just few hundred 1MB keys. Heap dump on OOME reveals 150+ instances of ContainerCommandRequestMessage, each with 16MB byte[].

      Steps to reproduce:

      1. Start Ozone cluster with 1 datanode
      2. Start Freon (5K keys of size 1MB)

      Result: OOME after a few hundred keys

      $ cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone
      $ docker-compose up -d
      $ docker-compose exec scm bash
      $ export HADOOP_OPTS='-XX:+HeapDumpOnOutOfMemoryError'
      $ ozone freon rk --numOfThreads 1 --numOfVolumes 1 --numOfBuckets 1 --replicationType RATIS --factor ONE --keySize 1048576 --numOfKeys 5120 --bufferSize 65536
      ...
      java.lang.OutOfMemoryError: Java heap space
      Dumping heap to java_pid289.hprof ...
      Heap dump file created [1456141975 bytes in 7.760 secs]
      

        Attachments

        1. profiler.png
          629 kB
          Attila Doroszlai

          Issue Links

            Activity

              People

              • Assignee:
                shashikant Shashikant Banerjee
                Reporter:
                adoroszlai Attila Doroszlai
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: