Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4478

Account for Kudu client memory in MemTracker

    Details

      Description

      The Kudu client may consume non-trivial amounts of memory and it is not accounted for in the query MemTracker, so it may be possible for the process to run out of memory.

      In particular, we need to consider the following in the KuduTableSink:

      1. Buffer space for write ops to be sent, which is 100MB by default and is configurable via a flag.
      2. Per-row errors observed by the client (before they are fetched and deleted by Impala). Each error contains a string and a copy of the row. The client API indicates that the error handling could overflow, i.e. that it is bounded, but the implementation does not yet limit the errors so this could be unbounded.

      We need to also understand whether there are any non-negligible memory allocations in the KuduScanNode.

        Attachments

          Activity

            People

            • Assignee:
              mjacobs Matthew Jacobs
              Reporter:
              mjacobs Matthew Jacobs
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: