Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-2238

Big DMS not flush under memory pressure

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.3.0
    • Fix Version/s: 1.7.0
    • Component/s: None
    • Labels:
      None
    • Environment:
      CentOS6.5 Linux 2.6.32-431
      Kudu1.3.0
      GitCommit 00813f96b9cb

      Description

      I have a table with many updates, its DMS consumes a lot memory and cause “Soft Memory Limit Exceed”. I check the /mem-trackers on the tablet server, one of its DMS consumes about 3G memory, but check the /maintenance-manager, its FlushDeltaMemStoresOp can only free 763B anchored memory and perf_improvement is 0. Is this normal? I know Kudu is not optimized for updates, but still confused why the DMS won’t be flushed under memory pressure.

      Infos from /mem-trackers:

      tablet-5941a8bb934e4686abd1bfff9e35c860 server none 3.00G 3.00G
      txn_tracker tablet-5941a8bb934e4686abd1bfff9e35c860 64.00M 0B 1.67M
      MemRowSet-339 tablet-5941a8bb934e4686abd1bfff9e35c860 none 265B 265B
      DeltaMemStores tablet-5941a8bb934e4686abd1bfff9e35c860 none 3.00G 3.00G

      Infos from /maintenance-manager

      FlushDeltaMemStoresOp(5941a8bb934e4686abd1bfff9e35c860) true 763B 511.15M 0

      The tablet 5941a8bb934e4686abd1bfff9e35c860 has 16 RowSets in total

      Some configs of MM:
      --enable_maintenance_manager=true
      --log_target_replay_size_mb=1024
      --maintenance_manager_history_size=8
      --maintenance_manager_num_threads=6
      --maintenance_manager_polling_interval_ms=50

        Attachments

        1. memory_consumed.png
          96 kB
          ZhangZhen
        2. memory_anchored.png
          31 kB
          ZhangZhen

          Activity

            People

            • Assignee:
              zhquake@gmail.com ZhangZhen
              Reporter:
              zhquake@gmail.com ZhangZhen
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: