Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5156

Drop VLOG level passed into Kudu client

    Details

    • Epic Color:
      ghx-label-2

      Description

      I'm running a concurrent stress workload with lots of small queries. By changing the impalad vlog level from 1 to 0, my QPS went from ~17qps to around 60qps.

      We should consider passing FLAGS_v - 1 into Kudu's vlog level, since we generally assume that VLOG=1 is allowed to be relatively noisy (eg logging thread start/stop, etc) whereas it seems like Impala's policy is that VLOG(1) should be fairly quiet.

        Activity

        Hide
        mjacobs Matthew Jacobs added a comment -

        commit 676a88f346c3dac57ffe0b5742a64ee7136b6bb1
        Author: Matthew Jacobs <mj@cloudera.com>
        Date: Tue Apr 4 13:36:01 2017 -0700

        IMPALA-5156: Drop VLOG level passed into Kudu client

        The Kudu logging is very verbose and all of the logging may
        affect performance. This change reduces the Kudu client log
        level to v-1, since Impala is typically run at v=1 and isn't
        too noisy.

        In some simple concurrency testing, Todd found that reducing
        the vlog level resulted in an increase in throughput from
        ~17 qps to 60qps.

        Change-Id: I6f160e20de37ac13887ccfebc5fb1d595fd05db1
        Reviewed-on: http://gerrit.cloudera.org:8080/6549
        Reviewed-by: Matthew Jacobs <mj@cloudera.com>
        Tested-by: Impala Public Jenkins

        Show
        mjacobs Matthew Jacobs added a comment - commit 676a88f346c3dac57ffe0b5742a64ee7136b6bb1 Author: Matthew Jacobs <mj@cloudera.com> Date: Tue Apr 4 13:36:01 2017 -0700 IMPALA-5156 : Drop VLOG level passed into Kudu client The Kudu logging is very verbose and all of the logging may affect performance. This change reduces the Kudu client log level to v-1, since Impala is typically run at v=1 and isn't too noisy. In some simple concurrency testing, Todd found that reducing the vlog level resulted in an increase in throughput from ~17 qps to 60qps. Change-Id: I6f160e20de37ac13887ccfebc5fb1d595fd05db1 Reviewed-on: http://gerrit.cloudera.org:8080/6549 Reviewed-by: Matthew Jacobs <mj@cloudera.com> Tested-by: Impala Public Jenkins

          People

          • Assignee:
            Unassigned
            Reporter:
            tlipcon Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development