HBase
  1. HBase
  2. HBASE-3455

Heap fragmentation in region server

    Details

    • Type: Brainstorming Brainstorming
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.90.1
    • Fix Version/s: 0.90.1
    • Component/s: Performance, regionserver
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Adds an allocation buffer local to each memstore which is used to combat heap fragmentation. Experimental in 0.90.1 - enable by toggling hbase.hregion.memstore.mslab.enabled. Enabled by default in 0.91

      Description

      Stop-the-world GC pauses have long been a problem in HBase. "Concurrent mode failures" can usually be tuned around by setting the initiating occupancy fraction low, but eventually the heap becomes fragmented and a promotion failure occurs.

      This JIRA is to do research/experiments about the heap fragmentation issue and possible solutions.

      1. with-kvallocs.png
        79 kB
        Todd Lipcon
      2. parse-fls-statistics.py
        0.9 kB
        Todd Lipcon
      3. mslab-6.txt
        27 kB
        Todd Lipcon
      4. mslab-5.txt
        26 kB
        Todd Lipcon
      5. mslab-4.txt
        23 kB
        Todd Lipcon
      6. mslab-3.txt
        23 kB
        Todd Lipcon
      7. mslab-2.txt
        22 kB
        Todd Lipcon
      8. mslab-1.txt
        17 kB
        Todd Lipcon
      9. icv-frag.png
        109 kB
        Todd Lipcon
      10. HBasefragmentation.pdf
        194 kB
        Todd Lipcon
      11. collapse-arrays.patch
        6 kB
        Todd Lipcon

        Activity

        Hide
        Ted Yu added a comment -

        I think this feature should be turned off automatically when region server uses small heap.

        Show
        Ted Yu added a comment - I think this feature should be turned off automatically when region server uses small heap.
        Hide
        Todd Lipcon added a comment -

        Yes, elimination of stop-the-world GC pauses for a lot of applications. Memory consumption likely to be about the same.

        If you have thousands of regions per server and a small heap, could end up being more memory usage though, currently.

        Show
        Todd Lipcon added a comment - Yes, elimination of stop-the-world GC pauses for a lot of applications. Memory consumption likely to be about the same. If you have thousands of regions per server and a small heap, could end up being more memory usage though, currently.
        Hide
        Otis Gospodnetic added a comment -

        In plain English, what's the end effect of this change? Elimination or minimalization of stop zee world GC pauses? Or maybe (also) lower memory consumption? Or...? Thanks.

        Show
        Otis Gospodnetic added a comment - In plain English, what's the end effect of this change? Elimination or minimalization of stop zee world GC pauses? Or maybe (also) lower memory consumption? Or...? Thanks.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #1737 (See https://hudson.apache.org/hudson/job/HBase-TRUNK/1737/)

        Show
        Hudson added a comment - Integrated in HBase-TRUNK #1737 (See https://hudson.apache.org/hudson/job/HBase-TRUNK/1737/ )
        Hide
        stack added a comment -

        +1 on commit. Its great.

        Show
        stack added a comment - +1 on commit. Its great.
        Hide
        Todd Lipcon added a comment -

        New rev takes Stack's suggestions:

        • better comment on the upsert code
        • actually enabled the test in memstore, since fixing the upsert code lets that test pass - this way if we accidentally start using MSLAB for upsert it'll fail.
        • MemStoreLAB now returns null for large allocs, and MemStore doesnt clone
        Show
        Todd Lipcon added a comment - New rev takes Stack's suggestions: better comment on the upsert code actually enabled the test in memstore, since fixing the upsert code lets that test pass - this way if we accidentally start using MSLAB for upsert it'll fail. MemStoreLAB now returns null for large allocs, and MemStore doesnt clone
        Hide
        stack added a comment -

        +1

        Minors below:

        On commit, I wonder if you should commit w/ it disabled on branch but enabled on trunk?

        In here:

        +    // Use the internalAdd method here since we (a) already have a lock
        +    // and (b) cannot safely use the MSLAB here.
        +    // See TestMemStore.DISABLED_testUpsertMSLAB
        

        ... its not safe because it would bloat memory? Or is it some thread 'safety' issue? Maybe clarify on commit?

        Should this class even be public?

        +public class MemStoreLAB {

        Why this?

        +    if (size > maxAlloc) {
        +      return new Allocation(new byte[size], 0);
        +    }
        

        On return we'll clone when we don't need to? Return null as flag that we're not to clone? (Caller is maybeCloneWithAllocator).

        We should open new issue for fixing upsert?

        Show
        stack added a comment - +1 Minors below: On commit, I wonder if you should commit w/ it disabled on branch but enabled on trunk? In here: + // Use the internalAdd method here since we (a) already have a lock + // and (b) cannot safely use the MSLAB here. + // See TestMemStore.DISABLED_testUpsertMSLAB ... its not safe because it would bloat memory? Or is it some thread 'safety' issue? Maybe clarify on commit? Should this class even be public? +public class MemStoreLAB { Why this? + if (size > maxAlloc) { + return new Allocation( new byte [size], 0); + } On return we'll clone when we don't need to? Return null as flag that we're not to clone? (Caller is maybeCloneWithAllocator). We should open new issue for fixing upsert?
        Hide
        Todd Lipcon added a comment -

        New version fixes ICV.

        I ran this one through the full test suite with MSLAB enabled, and it passed. (my previous test run had been with mslab disabled so it didn't show that ICV was massively broken!)

        Also includes a test case that triggers the pathological behavior.

        Show
        Todd Lipcon added a comment - New version fixes ICV. I ran this one through the full test suite with MSLAB enabled, and it passed. (my previous test run had been with mslab disabled so it didn't show that ICV was massively broken!) Also includes a test case that triggers the pathological behavior.
        Hide
        Todd Lipcon added a comment -

        Turns out there was a big bug in this which made upsert basically not work at all!

        New patch coming soon.

        Show
        Todd Lipcon added a comment - Turns out there was a big bug in this which made upsert basically not work at all! New patch coming soon.
        Hide
        ryan rawson added a comment -

        just had a look at the patch, it is looking good. nice minimal touch point w/memstore, and I am also happy to note the chunks are being dereferenced during rollover. The other CAS logic seems to make sense as well.

        Show
        ryan rawson added a comment - just had a look at the patch, it is looking good. nice minimal touch point w/memstore, and I am also happy to note the chunks are being dereferenced during rollover. The other CAS logic seems to make sense as well.
        Hide
        stack added a comment -

        +1 on commit to branch and trunk (fix licenses on commit). I think its important that this stuff get an airing as soon as possible (and its off by default).

        Show
        stack added a comment - +1 on commit to branch and trunk (fix licenses on commit). I think its important that this stuff get an airing as soon as possible (and its off by default).
        Hide
        Todd Lipcon added a comment -

        Fixes heap size test and removes the advanced tunables from hbase-default.xml

        Show
        Todd Lipcon added a comment - Fixes heap size test and removes the advanced tunables from hbase-default.xml
        Hide
        Ted Yu added a comment -

        hbase.hregion.memstore.mslab.chunksize appears twice in hbase-default.xml

        Show
        Ted Yu added a comment - hbase.hregion.memstore.mslab.chunksize appears twice in hbase-default.xml
        Hide
        Jonathan Gray added a comment -

        +1

        Show
        Jonathan Gray added a comment - +1
        Hide
        Todd Lipcon added a comment -

        Yea.. how about we put in the on/off switch in hbase-default, but leave the size switches undocumented except for in the source? (I have no idea what the best values are for those... could be very dependent on your JVM settings even)

        Ted also reminded me that some unit tests are failing (heapsize tests for Store for example), so we should run those before committing and fix up - I don't think I got them all right yet.

        Show
        Todd Lipcon added a comment - Yea.. how about we put in the on/off switch in hbase-default, but leave the size switches undocumented except for in the source? (I have no idea what the best values are for those... could be very dependent on your JVM settings even) Ted also reminded me that some unit tests are failing (heapsize tests for Store for example), so we should run those before committing and fix up - I don't think I got them all right yet.
        Hide
        Jonathan Gray added a comment -

        +1 on keeping exotic config out of hbase-default though i'm not sure just the turning on/off of this is that exotic. Definitely need some documentation/explanation somewhere tho.

        Show
        Jonathan Gray added a comment - +1 on keeping exotic config out of hbase-default though i'm not sure just the turning on/off of this is that exotic. Definitely need some documentation/explanation somewhere tho.
        Hide
        Jonathan Gray added a comment -

        I'm cool with 0.90.1 because it would be good to get this out there for people to experiment with sooner than later (and it's off by default, which I think it needs to be until we do more experimenting with more workloads in more environments). Seems like there's a lot of configuration params that could go into this too that may need tweaking in general and in different setups?

        Show
        Jonathan Gray added a comment - I'm cool with 0.90.1 because it would be good to get this out there for people to experiment with sooner than later (and it's off by default, which I think it needs to be until we do more experimenting with more workloads in more environments). Seems like there's a lot of configuration params that could go into this too that may need tweaking in general and in different setups?
        Hide
        stack added a comment -

        +1 on commit and on 0.90.1 (Fix license so its 2011 on commit, I don't know about exposing such exotic configs in hbase-default.xml but will not object to their presence). This is great Todd. Want to open new issue to figure the ICV issue?

        Show
        stack added a comment - +1 on commit and on 0.90.1 (Fix license so its 2011 on commit, I don't know about exposing such exotic configs in hbase-default.xml but will not object to their presence). This is great Todd. Want to open new issue to figure the ICV issue?
        Hide
        Todd Lipcon added a comment -

        Marking for 0.90.1 since the new feature is off-by-default. Happy to punt to 0.92 if others disagree.

        Show
        Todd Lipcon added a comment - Marking for 0.90.1 since the new feature is off-by-default. Happy to punt to 0.92 if others disagree.
        Hide
        Todd Lipcon added a comment -

        Oops, missed one test change before.

        This patch should be good to go. I tested under heavy load (95% write, 5% increment, large working set to trigger block cache churn) on a cluster for about 16 hours, no full GCs at all in 8G heap with CMS.

        But it's still off by default because of the worrisome theoretical possibilities with upsert.

        Show
        Todd Lipcon added a comment - Oops, missed one test change before. This patch should be good to go. I tested under heavy load (95% write, 5% increment, large working set to trigger block cache churn) on a cluster for about 16 hours, no full GCs at all in 8G heap with CMS. But it's still off by default because of the worrisome theoretical possibilities with upsert.
        Hide
        Todd Lipcon added a comment -

        New patch where this feature is off-by-default, enabled by setting hbase.hregion.memstore.mslab.enabled

        Show
        Todd Lipcon added a comment - New patch where this feature is off-by-default, enabled by setting hbase.hregion.memstore.mslab.enabled
        Hide
        Jonathan Gray added a comment -

        A slight aside, but in a heavy concurrency increment-only use case we are CPU bound with 80% or so being used in MemStore / CSLM operations. We are playing with different ways of optimizing the MemStore+Increments use case. Today, it takes between 3 and 4 CSLM operations for each increment. We should be able to get this down to 1 or 2, but in all likelihood this would be an increment-only optimization. Rather than modifying the original bytes in place, we are making a new byte[] and modifying the KV to point at the new byte[]. This keeps the modification atomic and prevents build-up (and costly removal) of KVs.

        Will file a JIRA for this.

        Show
        Jonathan Gray added a comment - A slight aside, but in a heavy concurrency increment-only use case we are CPU bound with 80% or so being used in MemStore / CSLM operations. We are playing with different ways of optimizing the MemStore+Increments use case. Today, it takes between 3 and 4 CSLM operations for each increment. We should be able to get this down to 1 or 2, but in all likelihood this would be an increment-only optimization. Rather than modifying the original bytes in place, we are making a new byte[] and modifying the KV to point at the new byte[]. This keeps the modification atomic and prevents build-up (and costly removal) of KVs. Will file a JIRA for this.
        Hide
        ryan rawson added a comment -

        we cannot replace the old value with the new one because there is no
        way to atomically do that.

        Show
        ryan rawson added a comment - we cannot replace the old value with the new one because there is no way to atomically do that.
        Hide
        Tatsuya Kawano added a comment -

        Just realized ICV means fixed-size KeyValue, right? These old and new KeyValues must have the exactly same key (except fixed-size timestamp) with a fixed-size value (Long value). Maybe we can just override the old value on the slab with the new value?

        Show
        Tatsuya Kawano added a comment - Just realized ICV means fixed-size KeyValue, right? These old and new KeyValues must have the exactly same key (except fixed-size timestamp) with a fixed-size value (Long value). Maybe we can just override the old value on the slab with the new value?
        Hide
        stack added a comment -

        @Ryan Your idea will only work if the slab is full of ICVs all with same coordinates, is that right? Or you thinking something different? What if we did give each ICV a 2MB slab (or less) and then when it overflowed, we'd let go of the old slab as you suggest?

        @Todd I don't think the rewrite of a slab would be that bad perf-wise

        @Tatsuya IIRC, we used to do iCVs this way but we'd end up flushing way too much; thats when the replacing old values was putting into effect.

        Show
        stack added a comment - @Ryan Your idea will only work if the slab is full of ICVs all with same coordinates, is that right? Or you thinking something different? What if we did give each ICV a 2MB slab (or less) and then when it overflowed, we'd let go of the old slab as you suggest? @Todd I don't think the rewrite of a slab would be that bad perf-wise @Tatsuya IIRC, we used to do iCVs this way but we'd end up flushing way too much; thats when the replacing old values was putting into effect.
        Hide
        ryan rawson added a comment -

        for ICVs, we might be force flushing due to the maximum hlog fairly
        'frequently' anyways. if we spill to the next slab, will we let the
        GC free up the previous slabs? maybe once we fill up an allocation we
        dereference the slab from the slab manager that way we can let GC do
        the hard work for us?

        Show
        ryan rawson added a comment - for ICVs, we might be force flushing due to the maximum hlog fairly 'frequently' anyways. if we spill to the next slab, will we let the GC free up the previous slabs? maybe once we fill up an allocation we dereference the slab from the slab manager that way we can let GC do the hard work for us?
        Hide
        Tatsuya Kawano added a comment -

        Maybe this is not an optimal solution but how about this? Forget about reclaiming individual free spaces from old cell values, but pretend them as live spaces and let Memstore to flush. Memstore flusher won't write old cell values to HFile as they are not referenced from the Memstore.

        You could get a stupid smaller HFiles if you do lots of ICVs, but a minor compaction will soon merge them into one moderate-sized HFile, so it's not going to be a problem.

        Show
        Tatsuya Kawano added a comment - Maybe this is not an optimal solution but how about this? Forget about reclaiming individual free spaces from old cell values, but pretend them as live spaces and let Memstore to flush. Memstore flusher won't write old cell values to HFile as they are not referenced from the Memstore. You could get a stupid smaller HFiles if you do lots of ICVs, but a minor compaction will soon merge them into one moderate-sized HFile, so it's not going to be a problem.
        Hide
        Todd Lipcon added a comment -

        As expected, when I turned on ICV, I got a full GC within a few minutes.

        Show
        Todd Lipcon added a comment - As expected, when I turned on ICV, I got a full GC within a few minutes.
        Hide
        Todd Lipcon added a comment -

        Here's a working version of the patch. Fails some test cases around ICV, probably will blow up with OOME under an ICV workload

        Show
        Todd Lipcon added a comment - Here's a working version of the patch. Fails some test cases around ICV, probably will blow up with OOME under an ICV workload
        Hide
        Todd Lipcon added a comment -

        There's one big flaw in this which I've now realized: upserts

        The issue is that, even after we upsert a KV and drop its reference from the KV set, we don't actually free any memory, since that KV's byte[] was just a pointer to a larger chunk. So assumedly there is at least one other KV pointing into that same chunk. Let's say a 2M chunk supports 20000 incrementable cells. If we increment 19999 of them, we still have one KV pointing to it which uses up 2MB of RAM.

        Not sure exactly how to solve this... will sleep on it a few days...

        One thought is that the Memstore can keep an IdentityHashMap<byte[], AtomicInteger> chunk->referred_size map around. When we upsert something (and hence remove its KV from kvset) we decrement the value in this map for the chunk it referred to. If the amount of referred-to data in a chunk falls below some threshold (say 50%) then we run some "compaction" code which relocates the remaining values to a new contiguous chunk. Performance impact of this might be too much, but maybe addressable with striping the counters or something.

        Show
        Todd Lipcon added a comment - There's one big flaw in this which I've now realized: upserts The issue is that, even after we upsert a KV and drop its reference from the KV set, we don't actually free any memory, since that KV's byte[] was just a pointer to a larger chunk. So assumedly there is at least one other KV pointing into that same chunk. Let's say a 2M chunk supports 20000 incrementable cells. If we increment 19999 of them, we still have one KV pointing to it which uses up 2MB of RAM. Not sure exactly how to solve this... will sleep on it a few days... One thought is that the Memstore can keep an IdentityHashMap<byte[], AtomicInteger> chunk->referred_size map around. When we upsert something (and hence remove its KV from kvset) we decrement the value in this map for the chunk it referred to. If the amount of referred-to data in a chunk falls below some threshold (say 50%) then we run some "compaction" code which relocates the remaining values to a new contiguous chunk. Performance impact of this might be too much, but maybe addressable with striping the counters or something.
        Hide
        stack added a comment -

        Should this wait notify rather than spin?

        +      while (!initialized) {
        +        Thread.yield();
        +      }
        

        Patch is small. Reasonable. Lets commit it!

        Show
        stack added a comment - Should this wait notify rather than spin? + while (!initialized) { + Thread .yield(); + } Patch is small. Reasonable. Lets commit it!
        Hide
        stack added a comment -

        Hotdog!

        Show
        stack added a comment - Hotdog!
        Hide
        Todd Lipcon added a comment -

        Hey cool, looks like it worked. None of my regionservers had any full GCs. Max chunk size is still around 700MB after the YCSB load, and I think total throughput actually went up as well.

        Show
        Todd Lipcon added a comment - Hey cool, looks like it worked. None of my regionservers had any full GCs. Max chunk size is still around 700MB after the YCSB load, and I think total throughput actually went up as well.
        Hide
        Todd Lipcon added a comment -

        here's a fairly simple patch that adds a moderately efficient (read: CAS-happy) allocation arena per memstore. When a KV gets added to the memstore it clones its byte array into the arena. Hopefully this means that when the memstore gets freed, it will be able to release large chunks of heap.

        Show
        Todd Lipcon added a comment - here's a fairly simple patch that adds a moderately efficient (read: CAS-happy) allocation arena per memstore. When a KV gets added to the memstore it clones its byte array into the arena. Hopefully this means that when the memstore gets freed, it will be able to release large chunks of heap.
        Hide
        Todd Lipcon added a comment -

        Yep, I think the KVs are all mixed up like you said.

        I did make some progress on G1 - trying to get them to include a couple patches which make it perform somewhat better than CMS, at least in my environment.

        Also just did a patch tonight which will try to collapse the byte arrays together into large ones, will run it overnight and get a graph.

        Show
        Todd Lipcon added a comment - Yep, I think the KVs are all mixed up like you said. I did make some progress on G1 - trying to get them to include a couple patches which make it perform somewhat better than CMS, at least in my environment. Also just did a patch tonight which will try to collapse the byte arrays together into large ones, will run it overnight and get a graph.
        Hide
        Tatsuya Kawano added a comment -

        Hi Todd. Thanks for your comment. I'm looking forwared to seeing the next report!

        So, you're going to debug G1GC itself? Wow, you're the memory guy. G1GC sounds promissing and I wonder why it doesn't work well for us. Maybe, KeyValues from different memstores are mixed up in all over the place so it can't release any region?

        Show
        Tatsuya Kawano added a comment - Hi Todd. Thanks for your comment. I'm looking forwared to seeing the next report! So, you're going to debug G1GC itself? Wow, you're the memory guy. G1GC sounds promissing and I wonder why it doesn't work well for us. Maybe, KeyValues from different memstores are mixed up in all over the place so it can't release any region?
        Hide
        Todd Lipcon added a comment -

        Hi Tatsuya. Thanks for the comments. We were actually discussing this the other day and this morning in IRC and came to similar conclusions. I'll try to write up a summary of the various solutions and some experiments we can try.

        I'm also experimenting again with G1GC - finally got a build environment for JDK7 so I can add some printfs and understand why it hasn't worked well for us yet.

        Show
        Todd Lipcon added a comment - Hi Tatsuya. Thanks for the comments. We were actually discussing this the other day and this morning in IRC and came to similar conclusions. I'll try to write up a summary of the various solutions and some experiments we can try. I'm also experimenting again with G1GC - finally got a build environment for JDK7 so I can add some printfs and understand why it hasn't worked well for us yet.
        Hide
        Tatsuya Kawano added a comment -

        Very nice report, Todd.

        Here is my thought on memstore, though, this is pure assumption and I need somebody to verify it.

        1. Puts creates KeyValues, and each of them has their own small byte array.
        2. Those KeyValues are stored in a memstore and survive for a few CMS GCs, get promoted to the old generation.
        3. The memstore eventually gets full and flushed. KeyValues and their small byte arrays on the old generation are GCed, resulting the fragmentation.

        To workaround this, we could use a larger (some MB?) byte array and allocate some portion of it to each kv. (Share one big array with a number of KeyValues instead of having small their own arrays.) When the memstore is flushed, we can wipe the contents out and reuse the array. In this way, those byte arrays will stay in the old generation forever and never bother the GC.

        Show
        Tatsuya Kawano added a comment - Very nice report, Todd. Here is my thought on memstore, though, this is pure assumption and I need somebody to verify it. 1. Puts creates KeyValues, and each of them has their own small byte array. 2. Those KeyValues are stored in a memstore and survive for a few CMS GCs, get promoted to the old generation. 3. The memstore eventually gets full and flushed. KeyValues and their small byte arrays on the old generation are GCed, resulting the fragmentation. To workaround this, we could use a larger (some MB?) byte array and allocate some portion of it to each kv. (Share one big array with a number of KeyValues instead of having small their own arrays.) When the memstore is flushed, we can wipe the contents out and reuse the array. In this way, those byte arrays will stay in the old generation forever and never bother the GC.
        Hide
        Todd Lipcon added a comment -

        For reference, the R commands I use to produce graphs look like:

        library(ggplot2)
        d <- read.table(file="/tmp/fls.txt", header=T)
        qplot(time, value, data=melt(subset(d, time>122000&time<127000), c("time"), measure.vars=c("free_space", "max_chunk")), facets=variable~., geom="line")
        

        And since JIRA screwed up formatting on those JVM options, repaste:

        -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -verbose:gc
        -Xloggc:$HBASE_HOME/logs/gc$(hostname)-hbase.log
        -XX:PrintCMSStatistics=1 -XX:PrintFLSStatistics=1
        
        Show
        Todd Lipcon added a comment - For reference, the R commands I use to produce graphs look like: library(ggplot2) d <- read.table(file= "/tmp/fls.txt" , header=T) qplot(time, value, data=melt(subset(d, time>122000&time<127000), c( "time" ), measure.vars=c( "free_space" , "max_chunk" )), facets=variable~., geom= "line" ) And since JIRA screwed up formatting on those JVM options, repaste: -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -verbose:gc -Xloggc:$HBASE_HOME/logs/gc$(hostname)-hbase.log -XX:PrintCMSStatistics=1 -XX:PrintFLSStatistics=1
        Hide
        Todd Lipcon added a comment -

        Here's the python script I use to convert the FLS statistics log into a TSV format for graphing.

        I was using the following flags with JVM 6u23:
        XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -verbose:gc -Xloggc:$HBASE_HOME/logs/gc$(hostname)-hbase.log
        -XX:PrintCMSStatistics=1 -XX:PrintFLSStatistics=1

        Might need to fiddle with the regex in the script if you use different flags or a different JVM release.

        Show
        Todd Lipcon added a comment - Here's the python script I use to convert the FLS statistics log into a TSV format for graphing. I was using the following flags with JVM 6u23: XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -verbose:gc -Xloggc:$HBASE_HOME/logs/gc $(hostname)-hbase.log -XX:PrintCMSStatistics=1 -XX:PrintFLSStatistics=1 Might need to fiddle with the regex in the script if you use different flags or a different JVM release.
        Hide
        stack added a comment -

        Upping to critical. I love this report of yours Todd.

        Show
        stack added a comment - Upping to critical. I love this report of yours Todd.
        Hide
        Todd Lipcon added a comment -

        Here's a PDF showing some graphs about heap fragmentation. These graphs indicate that memstore is way worse of a problem than block cache, though block cache churn can also introduce some fragmentation.

        Show
        Todd Lipcon added a comment - Here's a PDF showing some graphs about heap fragmentation. These graphs indicate that memstore is way worse of a problem than block cache, though block cache churn can also introduce some fragmentation.

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            18 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development