Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4566

Kudu client glog contention can cause timeouts

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0
    • Fix Version/s: Impala 2.8.0
    • Component/s: Backend
    • Labels:

      Description

      There appears to be significant contention in glog from the Kudu client threads under stressful workloads, we believe resulting in timeouts/failed queries.

      Kudu JIRA: https://issues.apache.org/jira/browse/KUDU-695

      Todd filed a glog upstream JIRA to address the contention. The Kudu client may need to be more careful with what is getting logged.

      In Impala, we can mitigate some of this by setting the Kudu log level separately from the Impala log level, and offering the ability to disable the Kudu client logging completely.

        Activity

        Hide
        twmarshall Thomas Tauber-Marshall added a comment -

        commit 2ae93b03ee00aff84e25db51813c9d1f5cb26b8b
        Author: Thomas Tauber-Marshall <tmarshall@cloudera.com>
        Date: Fri Dec 2 15:15:55 2016 -0800

        IMPALA-4566: Kudu client glog contention can cause timeouts

        Under stressful workloads, there appears to be significant contention
        in glog resulting from Kudu logging, causing timeouts and failed
        queries.

        An easy solution for now is to downgrade Kudu WARNINGs to INFOs as
        WARNINGs grab a lock to flush the log. This is appropriate as Kudu
        logs WARNINGs much more frequently than Impala and for things that
        Impala would normally consider INFO-level.

        Testing: Manually verified that the patch redirects Kudu WARNINGs
        to the INFO log. Not tested under stress to verify if this actually
        solves the contention problem.

        Change-Id: Ie535d89ec2525232d4f6a29dd44f51cd6e18a0d2
        Reviewed-on: http://gerrit.cloudera.org:8080/5334
        Reviewed-by: Matthew Jacobs <mj@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        twmarshall Thomas Tauber-Marshall added a comment - commit 2ae93b03ee00aff84e25db51813c9d1f5cb26b8b Author: Thomas Tauber-Marshall <tmarshall@cloudera.com> Date: Fri Dec 2 15:15:55 2016 -0800 IMPALA-4566 : Kudu client glog contention can cause timeouts Under stressful workloads, there appears to be significant contention in glog resulting from Kudu logging, causing timeouts and failed queries. An easy solution for now is to downgrade Kudu WARNINGs to INFOs as WARNINGs grab a lock to flush the log. This is appropriate as Kudu logs WARNINGs much more frequently than Impala and for things that Impala would normally consider INFO-level. Testing: Manually verified that the patch redirects Kudu WARNINGs to the INFO log. Not tested under stress to verify if this actually solves the contention problem. Change-Id: Ie535d89ec2525232d4f6a29dd44f51cd6e18a0d2 Reviewed-on: http://gerrit.cloudera.org:8080/5334 Reviewed-by: Matthew Jacobs <mj@cloudera.com> Tested-by: Internal Jenkins
        Hide
        jbapple Jim Apple added a comment -

        This is a bulk comment on all issues with Fix Version 2.8.0 that were resolved on or after 2016-12-09.

        2.8.0 was branched on December 9, with only two changes to master cherry-picked to the 2.8.0 release branch after that:

        https://github.com/apache/incubator-impala/commits/2.8.0

        Issues fixed after December 9 might not be fixed in 2.8.0. If you are the one who marked this issue Resolved, can you check to see if the patch is in 2.8.0 by using the link above? If the patch is not in 2.8.0, can you change the Fix Version to 2.9.0?

        Thank you!

        Show
        jbapple Jim Apple added a comment - This is a bulk comment on all issues with Fix Version 2.8.0 that were resolved on or after 2016-12-09. 2.8.0 was branched on December 9, with only two changes to master cherry-picked to the 2.8.0 release branch after that: https://github.com/apache/incubator-impala/commits/2.8.0 Issues fixed after December 9 might not be fixed in 2.8.0. If you are the one who marked this issue Resolved, can you check to see if the patch is in 2.8.0 by using the link above? If the patch is not in 2.8.0, can you change the Fix Version to 2.9.0? Thank you!

          People

          • Assignee:
            mjacobs Matthew Jacobs
            Reporter:
            mjacobs Matthew Jacobs
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development