Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5301

minicluster kudu needs mem limits set

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Infrastructure
    • Labels:
    • Epic Color:
      ghx-label-1

      Description

      The minicluster runs Kudu without an explicit memory limit, and Kudu defaults to assuming it can use 80% of system memory. We should set a limit to avoid overcommitting system memory.

      We need to keep the value low because aws test nodes (jenkins.impala.io) are small, see IMPALA-5297. I tested a limit of 1gb but that wasn't enough – TestKuduMemLimits failed. We may try something a bit larger, or perhaps find other ways to reduce the requirements of minicluster tests.

        Activity

        Hide
        laszlog Laszlo Gaal added a comment -

        Matthew Jacobs, I saw actual test failures caused by this.

        I was running exhaustive-mode tests on an AWS c3.4xlarge instance (16 CPUs, 30 GB memory) and one of the tablet servers was regularly OOM-killed. I seem to remember each tablet server (and there are 3 of them in the minicluster) running with an 8 GB memory limit. Unfortunately I don't recall if it was a command-line parameter or a number I saw on a status page.

        Show
        laszlog Laszlo Gaal added a comment - Matthew Jacobs , I saw actual test failures caused by this. I was running exhaustive-mode tests on an AWS c3.4xlarge instance (16 CPUs, 30 GB memory) and one of the tablet servers was regularly OOM-killed. I seem to remember each tablet server (and there are 3 of them in the minicluster) running with an 8 GB memory limit. Unfortunately I don't recall if it was a command-line parameter or a number I saw on a status page.
        Hide
        mjacobs Matthew Jacobs added a comment -

        I'm testing a potential fix now: https://gerrit.cloudera.org/#/c/6844/

        Show
        mjacobs Matthew Jacobs added a comment - I'm testing a potential fix now: https://gerrit.cloudera.org/#/c/6844/
        Hide
        mjacobs Matthew Jacobs added a comment -

        commit d04f96b990c18ed17c82c0151978b56f936f0574
        Author: Matthew Jacobs <mj@cloudera.com>
        Date: Wed May 10 11:10:21 2017 -0700

        IMPALA-5301: Set Kudu minicluster memory limit

        By default, Kudu assumes it has 80% of system memory which
        is far too high for the minicluster. This sets a mem limit
        of 2gb and lowers the limit of the block cache. These values
        were tested on a gerrit-verify-dryrun job as well as an
        exhaustive run.

        This patch also simplifies TestKuduMemLimits which was
        unnecessarily creating a large table during test execution.

        Change-Id: I7fd7e1cd9dc781aaa672a2c68c845cb57ec885d5
        Reviewed-on: http://gerrit.cloudera.org:8080/6844
        Reviewed-by: Todd Lipcon <todd@apache.org>
        Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
        Tested-by: Impala Public Jenkins

        Show
        mjacobs Matthew Jacobs added a comment - commit d04f96b990c18ed17c82c0151978b56f936f0574 Author: Matthew Jacobs <mj@cloudera.com> Date: Wed May 10 11:10:21 2017 -0700 IMPALA-5301 : Set Kudu minicluster memory limit By default, Kudu assumes it has 80% of system memory which is far too high for the minicluster. This sets a mem limit of 2gb and lowers the limit of the block cache. These values were tested on a gerrit-verify-dryrun job as well as an exhaustive run. This patch also simplifies TestKuduMemLimits which was unnecessarily creating a large table during test execution. Change-Id: I7fd7e1cd9dc781aaa672a2c68c845cb57ec885d5 Reviewed-on: http://gerrit.cloudera.org:8080/6844 Reviewed-by: Todd Lipcon <todd@apache.org> Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Impala Public Jenkins

          People

          • Assignee:
            Unassigned
            Reporter:
            mjacobs Matthew Jacobs
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development