• Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.90.0
    • Fix Version/s: 0.90.0
    • Component/s: master
    • Labels:
    • Hadoop Flags:


      In testing a RS failure under heavy increment workload I ran into an OOME when the master was splitting the logs.

      In this test case, I have exactly 136 bytes per log entry in all the logs, and the logs are all around 66-74MB). With a batch size of 3 logs, this means the master is loading about 500K-600K edits per log file. Each edit ends up creating 3 byte[] objects, the references for which are each 8 bytes of RAM, so we have 160 (136+8*3) bytes per edit used by the byte[]. For each edit we also allocate a bunch of other objects: one HLog$Entry, one WALEdit, one ArrayList, one LinkedList$Entry, one HLogKey, and one KeyValue. Overall this works out to 400 bytes of overhead per edit. So, with the default settings on this fairly average workload, the 1.5M log entries takes about 770MB of RAM. Since I had a few log files that were a bit larger (around 90MB) it exceeded 1GB of RAM and I got an OOME.

      For one, the 400 bytes per edit overhead is pretty bad, and we could probably be a lot more efficient. For two, we should actually account this rather than simply having a configurable "batch size" in the master.

      I think this is a blocker because I'm running with fairly default configs here and just killing one RS made the cluster fall over due to master OOME.

      1. sizes.png
        81 kB
        Todd Lipcon
      2. hbase-3323.txt
        27 kB
        Todd Lipcon
      3. hbase-3323.txt
        60 kB
        Todd Lipcon
      4. hbase-3323.txt
        61 kB
        Todd Lipcon
      5. hbase-3323.4.txt
        71 kB
        Todd Lipcon
      6. hbase-3323.5.txt
        71 kB
        Todd Lipcon
      7. hbase-3323.6.txt
        71 kB

        Issue Links


          Todd Lipcon created issue -
          Todd Lipcon made changes -
          Field Original Value New Value
          Attachment sizes.png [ 12465886 ]
          Todd Lipcon made changes -
          Link This issue relates to HBASE-3325 [ HBASE-3325 ]
          Cosmin Lehene made changes -
          Link This issue relates to HBASE-1364 [ HBASE-1364 ]
          Todd Lipcon made changes -
          Assignee Todd Lipcon [ tlipcon ]
          Todd Lipcon made changes -
          Attachment hbase-3323.txt [ 12465969 ]
          Todd Lipcon made changes -
          Attachment hbase-3323.txt [ 12466032 ]
          Todd Lipcon made changes -
          Attachment hbase-3323.txt [ 12466034 ]
          Todd Lipcon made changes -
          Attachment hbase-3323.4.txt [ 12466089 ]
          Todd Lipcon made changes -
          Attachment hbase-3323.5.txt [ 12466188 ]
          stack made changes -
          Attachment hbase-3323.6.txt [ 12466657 ]
          stack made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]


            • Assignee:
              Todd Lipcon
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              1 Start watching this issue


              • Created: