Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5156

Drop VLOG level passed into Kudu client

        Details

        • Epic Color:
          ghx-label-2

          Description

          I'm running a concurrent stress workload with lots of small queries. By changing the impalad vlog level from 1 to 0, my QPS went from ~17qps to around 60qps.

          We should consider passing FLAGS_v - 1 into Kudu's vlog level, since we generally assume that VLOG=1 is allowed to be relatively noisy (eg logging thread start/stop, etc) whereas it seems like Impala's policy is that VLOG(1) should be fairly quiet.

            Activity

            Hide
            Permalink
            mjacobs Matthew Jacobs added a comment -

            commit 676a88f346c3dac57ffe0b5742a64ee7136b6bb1
            Author: Matthew Jacobs <mj@cloudera.com>
            Date: Tue Apr 4 13:36:01 2017 -0700

            IMPALA-5156: Drop VLOG level passed into Kudu client

            The Kudu logging is very verbose and all of the logging may
            affect performance. This change reduces the Kudu client log
            level to v-1, since Impala is typically run at v=1 and isn't
            too noisy.

            In some simple concurrency testing, Todd found that reducing
            the vlog level resulted in an increase in throughput from
            ~17 qps to 60qps.

            Change-Id: I6f160e20de37ac13887ccfebc5fb1d595fd05db1
            Reviewed-on: http://gerrit.cloudera.org:8080/6549
            Reviewed-by: Matthew Jacobs <mj@cloudera.com>
            Tested-by: Impala Public Jenkins

            Show
            mjacobs Matthew Jacobs added a comment - commit 676a88f346c3dac57ffe0b5742a64ee7136b6bb1 Author: Matthew Jacobs <mj@cloudera.com> Date: Tue Apr 4 13:36:01 2017 -0700 IMPALA-5156 : Drop VLOG level passed into Kudu client The Kudu logging is very verbose and all of the logging may affect performance. This change reduces the Kudu client log level to v-1, since Impala is typically run at v=1 and isn't too noisy. In some simple concurrency testing, Todd found that reducing the vlog level resulted in an increase in throughput from ~17 qps to 60qps. Change-Id: I6f160e20de37ac13887ccfebc5fb1d595fd05db1 Reviewed-on: http://gerrit.cloudera.org:8080/6549 Reviewed-by: Matthew Jacobs <mj@cloudera.com> Tested-by: Impala Public Jenkins

              People

              • Assignee:
                Unassigned
                Reporter:
                tlipcon Todd Lipcon
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development