Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
1.1.0
-
None
Description
The /mem-trackers page shows hierarchical memory consumption according to Kudu's memtrackers while the /memz page dumps tcmalloc-specific statistics. Unlike other memtrackers, the root tracker's "consumption" is determined by asking tcmalloc for the "generic.current_allocated_bytes" stat, so the value ought to be the same as what's reported by /memz. A cluster under heavy load had a kudu master whose root tracker consumption was ~500MB while /memz showed a consumption of ~6GB. After some digging, I think this discrepancy is due to the interplay between regular trackers and "observer" trackers.
First, some background. Regular memory trackers have a consumption counter that is updated when memory is consumed or released. Observer trackers (of which there is only one: the root tracker) also have a "consumption function". At predetermined times, the function is invoked and the return value is stored in the consumption counter. These predetermined times are the MemTracker Consume(), Release(), and GcMemory() functions. As alluded to earlier, memory tracking is hierarchical: when MemTracker::Consume() is invoked, all of this tracker's ancestors are also updated, all the way up to the root. One would expect this hierarchical update to happen by the way of Consume(), but actually, each ancestor's consumption counter is updated directly. This is profound for the root tracker: unless Consume() or Release() are invoked directly on the root tracker, its consumption function is only invoked on GcMemory(). I don't know why it works this way; perhaps it's an optimization to prevent the consumption function from being invoked too often.
So, why the discrepancy? This cluster is under heavy load, which means the master is seeing some action (most of the load is on the tservers). Normally the master's tcmalloc would release memory to the system after 128MB of MemTracker::Release(), but perhaps these allocations are all "incidental" and thus untracked by Kudu, so tcmalloc isn't releasing them. Meanwhile, there's no "real" load on the master (i.e. transactions, or something else that would check memory limits), so GcMemory() isn't invoked, and the root tracker's consumption is never reset from tcmalloc.
The main question is whether this is a cosmetic issue or a real one. A server under real load (e.g. a tserver) wouldn't experience this. As for the master, well, it's not doing much memory limit checking of its own, so the discrepancy shouldn't affect ongoing operations.
Nevertheless, this should be fixed.
Attachments
Issue Links
- duplicates
-
-
- Resolved
-