Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-2231

"materializing_iterator_do_pushdown=true" cause simple query slow

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.0, 1.5.0, 1.6.0
    • Fix Version/s: 1.5.1, 1.7.0, 1.6.1
    • Component/s: master, tserver
    • Labels:
      None
    • Environment:

      Description

      I ran the following SQL again and again
      while refresh 8050/scans page at the same time.

      sql:

      select count(xx_id),count(yy_id),count(time) from  test_table  where event_id =29983; 
      

      "Cells read from disk" is much more greater then table size when materializing_iterator_do_pushdown = true (default).

      after setting materializing_iterator_do_pushdown = false
      "Cells read from disk" reduced to some reasonable value (close to table size)
      and the sql run faster.

      here's detail:

      table under test:

      CREATE TABLE rawdata.test_table (
        day INT NOT NULL ENCODING BIT_SHUFFLE COMPRESSION DEFAULT_COMPRESSION,
        user_id BIGINT NOT NULL ENCODING BIT_SHUFFLE COMPRESSION DEFAULT_COMPRESSION,
        time TIMESTAMP NOT NULL ENCODING BIT_SHUFFLE COMPRESSION DEFAULT_COMPRESSION,
        event_id INT NULL ENCODING BIT_SHUFFLE COMPRESSION DEFAULT_COMPRESSION,
        distinct_id STRING NULL ENCODING DICT_ENCODING COMPRESSION DEFAULT_COMPRESSION,
        ...
        ...  other fields ...
        ...
        PRIMARY KEY (day, user_id, time, _offset)
      )
      PARTITION BY HASH (user_id) PARTITIONS 9
      STORED AS KUDU
      TBLPROPERTIES ( ... );
      

      table size (select count(1) from test_table) : 19510709

      CASE 1, materializing_iterator_do_pushdown = true
      756ACA6F105F0905EBCB79B940FFCE86.jpg

      CASE 2, materializing_iterator_do_pushdown = false (sql ran faster)
      F8C604537B8E921DDCCA78995DC11BDA.jpg

      it looks like kudu scan table multiple times for the simple sql caused by some silly bug.

        Attachments

        1. screenshot-1.png
          69 kB
          Fu Lili
        2. F8C604537B8E921DDCCA78995DC11BDA.jpg
          167 kB
          DawnZhang
        3. 756ACA6F105F0905EBCB79B940FFCE86.jpg
          243 kB
          DawnZhang

          Activity

            People

            • Assignee:
              danburkert Dan Burkert
              Reporter:
              dawn110110 DawnZhang
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: