Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4565

Incorrect results under ASAN with mt_dop > 0 and old aggs/joins

    Details

      Description

      There are 7300 rows in alltypes but count returns varying numbers.

      [tarmstrong-box.ca.cloudera.com:21000] > use functional_parquet; set mt_dop=1; select count(*) from alltypes; summary;
      Query: use functional_parquet
      MT_DOP set to 1
      Query: select count(*) from alltypes
      Query submitted at: 2016-11-30 22:31:17 (Coordinator: http://tarmstrong-box:25000)
      Query progress can be monitored at: http://tarmstrong-box:25000/query_plan?query_id=3346df982e7ba933:b9cbaacf00000000
      +----------+
      | count(*) |
      +----------+
      | 900      |
      +----------+
      Fetched 1 row(s) in 0.13s
      +--------------+--------+----------+----------+-------+------------+----------+---------------+-----------------------------+
      | Operator     | #Hosts | Avg Time | Max Time | #Rows | Est. #Rows | Peak Mem | Est. Peak Mem | Detail                      |
      +--------------+--------+----------+----------+-------+------------+----------+---------------+-----------------------------+
      | 03:AGGREGATE | 1      | 762.35us | 762.35us | 1     | 1          | 8.04 MB  | -1 B          | FINALIZE                    |
      | 02:EXCHANGE  | 1      | 56.20us  | 56.20us  | 3     | 1          | 0 B      | -1 B          | UNPARTITIONED               |
      | 01:AGGREGATE | 3      | 541.97us | 571.22us | 3     | 1          | 8.02 MB  | 10.00 MB      |                             |
      | 00:SCAN HDFS | 3      | 6.00ms   | 7.60ms   | 900   | 7.30K      | 16.00 KB | 0 B           | functional_parquet.alltypes |
      +--------------+--------+----------+----------+-------+------------+----------+---------------+-----------------------------+
      

        Activity

        Hide
        tarmstrong Tim Armstrong added a comment -

        IMPALA-4564,IMPALA-4565: mt_dop fixes for old aggs and joins

        Fix a test bug where we need to skip nested types tests for the old aggs
        and joins.

        Fix a product bug where *eos is not initialised by the MT scan node.
        This causes incorrect results when the calling ExecNode does not
        initialise the eos variable, e.g. the sort node and the old agg and join
        nodes.

        Testing:
        Added a test that reproduces the incorrect results with the sort node
        when run under ASAN

        Tested the mt_dop tests locally with old aggs and joins to ensure they
        pass.

        Change-Id: I48c50c8aa0c23710eb099fba252bc3c0cb74b313
        Reviewed-on: http://gerrit.cloudera.org:8080/5302
        Reviewed-by: Alex Behm <alex.behm@cloudera.com>
        Tested-by: Impala Public Jenkins

        Show
        tarmstrong Tim Armstrong added a comment - IMPALA-4564 , IMPALA-4565 : mt_dop fixes for old aggs and joins Fix a test bug where we need to skip nested types tests for the old aggs and joins. Fix a product bug where *eos is not initialised by the MT scan node. This causes incorrect results when the calling ExecNode does not initialise the eos variable, e.g. the sort node and the old agg and join nodes. Testing: Added a test that reproduces the incorrect results with the sort node when run under ASAN Tested the mt_dop tests locally with old aggs and joins to ensure they pass. Change-Id: I48c50c8aa0c23710eb099fba252bc3c0cb74b313 Reviewed-on: http://gerrit.cloudera.org:8080/5302 Reviewed-by: Alex Behm <alex.behm@cloudera.com> Tested-by: Impala Public Jenkins
        Hide
        tarmstrong Tim Armstrong added a comment -

        It's also reproducible with sorts under ASAN, e.g. the following returns only 900 rows:

         use functional_parquet; set mt_dop=1; select id from alltypes order by id; summary
        
        Show
        tarmstrong Tim Armstrong added a comment - It's also reproducible with sorts under ASAN, e.g. the following returns only 900 rows: use functional_parquet; set mt_dop=1; select id from alltypes order by id; summary

          People

          • Assignee:
            tarmstrong Tim Armstrong
            Reporter:
            tarmstrong Tim Armstrong
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development