Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0
    • Fix Version/s: Impala 2.8.0
    • Component/s: Backend
    • Labels:
      None

      Description

      We discovered that the current kudu client defaults in the kudu table sink are causing a large number of queries to fail. The current mutation buffer size is 100mb, which is too large and results in too much pressure on kudu. By decreasing the default value of this flag, we found our simple tests to be successful. This change only involves changing the default value of the flag, no new logic.

        Activity

        Hide
        mjacobs Matthew Jacobs added a comment -

        Kudu JIRA to fix this underlying issue related to backpressure: https://issues.apache.org/jira/browse/KUDU-1587

        Show
        mjacobs Matthew Jacobs added a comment - Kudu JIRA to fix this underlying issue related to backpressure: https://issues.apache.org/jira/browse/KUDU-1587
        Hide
        mjacobs Matthew Jacobs added a comment -

        commit 4fa9270e647b9c097295dcc13d97136cca3139ad
        Author: Matthew Jacobs <mj@cloudera.com>
        Date: Thu Dec 8 20:32:45 2016 -0800

        IMPALA-4633: Change broken gflag default for Kudu client mem

        We discovered that the current Kudu client defaults in the
        KuduTableSink are causing a large number of queries to
        timeout, failing the query. The current default value of the
        'mutation buffer size' is 100MB which results in higher
        write throughput than Kudu can currently handle on large
        clusters. By decreasing the value of this flag, more RPCs
        will be sent for the same amount of data, i.e. throttling
        the load on Kudu. We found tests to be more successful on
        200 nodes with a 10MB buffer size than the previous 100MB
        value where most queries couldn't complete due to timeouts.
        These queries were not timing out with the 10MB value. This
        appears to work well on 9 node stress tests as well.

        Change-Id: I0b3544f9a93c82e347f6e97540d6b561c30d09fd
        Reviewed-on: http://gerrit.cloudera.org:8080/5503
        Reviewed-by: Dan Hecht <dhecht@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        mjacobs Matthew Jacobs added a comment - commit 4fa9270e647b9c097295dcc13d97136cca3139ad Author: Matthew Jacobs <mj@cloudera.com> Date: Thu Dec 8 20:32:45 2016 -0800 IMPALA-4633 : Change broken gflag default for Kudu client mem We discovered that the current Kudu client defaults in the KuduTableSink are causing a large number of queries to timeout, failing the query. The current default value of the 'mutation buffer size' is 100MB which results in higher write throughput than Kudu can currently handle on large clusters. By decreasing the value of this flag, more RPCs will be sent for the same amount of data, i.e. throttling the load on Kudu. We found tests to be more successful on 200 nodes with a 10MB buffer size than the previous 100MB value where most queries couldn't complete due to timeouts. These queries were not timing out with the 10MB value. This appears to work well on 9 node stress tests as well. Change-Id: I0b3544f9a93c82e347f6e97540d6b561c30d09fd Reviewed-on: http://gerrit.cloudera.org:8080/5503 Reviewed-by: Dan Hecht <dhecht@cloudera.com> Tested-by: Internal Jenkins
        Hide
        jbapple Jim Apple added a comment -

        This is a bulk comment on all issues with Fix Version 2.8.0 that were resolved on or after 2016-12-09.

        2.8.0 was branched on December 9, with only two changes to master cherry-picked to the 2.8.0 release branch after that:

        https://github.com/apache/incubator-impala/commits/2.8.0

        Issues fixed after December 9 might not be fixed in 2.8.0. If you are the one who marked this issue Resolved, can you check to see if the patch is in 2.8.0 by using the link above? If the patch is not in 2.8.0, can you change the Fix Version to 2.9.0?

        Thank you!

        Show
        jbapple Jim Apple added a comment - This is a bulk comment on all issues with Fix Version 2.8.0 that were resolved on or after 2016-12-09. 2.8.0 was branched on December 9, with only two changes to master cherry-picked to the 2.8.0 release branch after that: https://github.com/apache/incubator-impala/commits/2.8.0 Issues fixed after December 9 might not be fixed in 2.8.0. If you are the one who marked this issue Resolved, can you check to see if the patch is in 2.8.0 by using the link above? If the patch is not in 2.8.0, can you change the Fix Version to 2.9.0? Thank you!

          People

          • Assignee:
            mjacobs Matthew Jacobs
            Reporter:
            mjacobs Matthew Jacobs
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development