Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4478

Account for Kudu client memory in MemTracker

    Details

      Description

      The Kudu client may consume non-trivial amounts of memory and it is not accounted for in the query MemTracker, so it may be possible for the process to run out of memory.

      In particular, we need to consider the following in the KuduTableSink:

      1. Buffer space for write ops to be sent, which is 100MB by default and is configurable via a flag.
      2. Per-row errors observed by the client (before they are fetched and deleted by Impala). Each error contains a string and a copy of the row. The client API indicates that the error handling could overflow, i.e. that it is bounded, but the implementation does not yet limit the errors so this could be unbounded.

      We need to also understand whether there are any non-negligible memory allocations in the KuduScanNode.

        Activity

        Hide
        mjacobs Matthew Jacobs added a comment -

        Marking this as a blocker because I found a case where an Impalad was killed by the OOM killer, almost positive it's due to this.

        Show
        mjacobs Matthew Jacobs added a comment - Marking this as a blocker because I found a case where an Impalad was killed by the OOM killer, almost positive it's due to this.
        Hide
        mjacobs Matthew Jacobs added a comment -

        Closing the blocker JIRA, though some incremental improvements could be considered in the future (we can file new JIRAs if it becomes apparent this below isn't working well enough).

        commit 7bfa6e3e3f20d741f36e713ea62f9324e5739ea0
        Author: Matthew Jacobs <mj@cloudera.com>
        Date: Mon Nov 14 15:31:38 2016 -0800

        IMPALA-4478: Initial Kudu client mem tracking for sink

        The Kudu client allocates memory which is not tracked by
        Impala. There are several sources, but the most significant
        is the memory allocated by the KuduSession on the write
        path. This can be >100MB, so it is important to track to
        avoid OOM.

        Moving forward, we should have a better way to track Kudu
        client memory, but for now we must at least handle this
        potentially problematic case.

        This changes the KuduTableSink to consume 200MB which should
        be enough for the 100MB write mutation buffer as well as
        100MB worth of errors buffered in the client before Impala
        takes ownership of them (and deletes them). This is left as
        a flag because it may turn out to be too high for some users
        and too low for others. When we have better support from
        Kudu (including KUDU-1752), we should simplify this.

        TODO: Handle DML w/ small or known resource requirements
        (e.g. VALUES specified or query has LIMIT) specially to
        avoid over-consumption.

        Testing: Have verified acceptable behavior in the stress
        test with a simple workload containing DML statements of
        moderate cardinality.

        Change-Id: I47f17a81e4362ab490019382fedc66c25f07080a
        Reviewed-on: http://gerrit.cloudera.org:8080/5152
        Reviewed-by: Matthew Jacobs <mj@cloudera.com>
        Reviewed-by: Dan Hecht <dhecht@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        mjacobs Matthew Jacobs added a comment - Closing the blocker JIRA, though some incremental improvements could be considered in the future (we can file new JIRAs if it becomes apparent this below isn't working well enough). commit 7bfa6e3e3f20d741f36e713ea62f9324e5739ea0 Author: Matthew Jacobs <mj@cloudera.com> Date: Mon Nov 14 15:31:38 2016 -0800 IMPALA-4478 : Initial Kudu client mem tracking for sink The Kudu client allocates memory which is not tracked by Impala. There are several sources, but the most significant is the memory allocated by the KuduSession on the write path. This can be >100MB, so it is important to track to avoid OOM. Moving forward, we should have a better way to track Kudu client memory, but for now we must at least handle this potentially problematic case. This changes the KuduTableSink to consume 200MB which should be enough for the 100MB write mutation buffer as well as 100MB worth of errors buffered in the client before Impala takes ownership of them (and deletes them). This is left as a flag because it may turn out to be too high for some users and too low for others. When we have better support from Kudu (including KUDU-1752 ), we should simplify this. TODO: Handle DML w/ small or known resource requirements (e.g. VALUES specified or query has LIMIT) specially to avoid over-consumption. Testing: Have verified acceptable behavior in the stress test with a simple workload containing DML statements of moderate cardinality. Change-Id: I47f17a81e4362ab490019382fedc66c25f07080a Reviewed-on: http://gerrit.cloudera.org:8080/5152 Reviewed-by: Matthew Jacobs <mj@cloudera.com> Reviewed-by: Dan Hecht <dhecht@cloudera.com> Tested-by: Internal Jenkins

          People

          • Assignee:
            mjacobs Matthew Jacobs
            Reporter:
            mjacobs Matthew Jacobs
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development