Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-3197

Tablet keeps all history schemas in memory may result in high memory consumption

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 1.12.0
    • None
    • tablet
    • None

    Description

      In case of high frequency of updating table, memory consumption of kudu-tserver may be very high, and the memory in not tracked in the memory page. 

      This is the memory usage of a tablet, the memory consumption of tablet-xxx‘s peak is 3.6G, but none of its' childrens' memory can reach.

      So I use pprof to get the heap sampling. The tserver started for long but the memory is still consuming by TabletBootstrap:PlayAlterSchemaRequest. 

      I change the `old_schemas_` in tablet_metadata.h to a fixed size vector,

        // Previous values of 'schema_'.
      // These are currently kept alive forever, under the assumption that
      // a given tablet won't have thousands of "alter table" calls.
      // They are kept alive so that callers of schema() don't need to
      // worry about reference counting or locking.
      std::vector<Schema*> old_schemas_;

      The heap sampling then becomes

      So, to make application layer more flexible, it could be better to make the size of the old_schemas configurable.
       

      Attachments

        1. screenshot-1.png
          162 kB
          wangningito
        2. image-2020-12-02-19-59-46-733.png
          162 kB
          wangningito
        3. image-2020-09-25-15-05-44-948.png
          97 kB
          wangningito
        4. image-2020-09-25-14-49-30-913.png
          162 kB
          wangningito
        5. image-2020-09-25-14-45-33-402.png
          229 kB
          wangningito

        Activity

          People

            wangning wangningito
            wangning wangningito
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 50m
                50m