Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Infrastructure
    • Labels:
      None

      Description

      Since CRUD queries for Impala/Kudu will have side-effects, the random query generator framework's query execution path needs to change to support CRUD. It needs to ensure the database under test doesn't end up so polluted that queries are no longer useful. Examples here including a DELETE that deletes all the rows; UPSERT/UPDATE that reduces variability in a column; INSERTs that make blow up the size of the database; etc.

      The query execution also needs to consider CRUD validation. This tasks is about laying the groundwork for that.

        Activity

        Show
        mikesbrown Michael Brown added a comment - https://gerrit.cloudera.org/#/c/5387/
        Hide
        mikesbrown Michael Brown added a comment -
        commit 54665120cbf4f242688659941ac50bacc8c221c8
        Author: Michael Brown <mikeb@cloudera.com>
        Date:   Thu Dec 1 09:27:43 2016 -0800
        
            IMPALA-4355: random query generator: modify statement execution flow to support DML
        
            - Rework the discrepancy searcher to run DML statements. We do this by
              using the query profile to choose a table, copy that table, and
              generate a statement that will INSERT into that copy. We chose a slow
              copy over other methods because INSERTing into a copy is a more
              reliable test that prevents table sizes from getting out of hand or
              time-consuming replay to reproduce a particular statement.
        
            - Introduce a statement generator stub. The real generator work is
              tracked in IMPALA-4351 and IMPALA-4353. Here we simply generate a
              basic INSERT INTO ... VALUES statement to make sure our general query
              execution flow is working.
        
            - Add query profile stub for DML statements (INSERT-only at this time).
              Since we'll want INSERT INTO ... SELECT very soon, this inherits from
              DefaultProfile. Also add building blocks for choosing random
              statements in the DefaultProfile.
        
            - Improve the concept of an "execution mode" and add new modes. Before,
              we had "RAW", "CREATE_TABLE_AS", and "CREATE_VIEW_AS". The idea here
              is that some random SELECT queries could be generated as "CREATE
              TABLE|VIEW AS" at execution time, based on weights in the query
              profile. First, we remove the use of raw string literals for this,
              since raw string literals can be error-prone, and introduce a
              StatementExecutionMode class to contain a namespace for the enumerated
              statement execution modes. Second, we introduce a couple new execution
              modes. The first is DML_SETUP: this is a DML statement that needs to
              be run in both the test and reference databases concurrently. For our
              purposes, it's the INSERT ... SELECT that copies data from the chosen
              random table into the table copy. The second is DML_TEST: this is a
              randomly-generated DML statement.
        
            - Switch to using absolute imports in many places. There was a mix of
              absolute and relative imports happening here, and they were causing
              problems, especially when comparing data types. In Python,
              <class 'db_types.Int'> != <class 'tests.comparison.db_types.Int'>.
              Using
                from __future__ import absolute_import
              didn't seem to catch the relative import usage anyway, so I haven't
              employed that.
        
            - Rename some, but not nearly all, names from "query" to "statement".
              Doing this is a rather large undertaking leading to much larger diffs
              and testing (IMPALA-4602).
        
            - Fix a handful of flake8 warnings. There are a bunch that went unfixed
              for over- and under-indentation.
        
            - Testing
              o ./discrepancy_searcher.py runs with and without --explain-only, and
              with --profile default and --profile dmlonly. For tpch_kudu data, it
              seems sufficient to use a --timeout of about 300.
              o Leopard run to make sure standard SELECT-only generation still works
              o Generated random stress queries locally
              o Generated random data locally
        
            Change-Id: Ia4c63a2223185d0e056cc5713796772e5d1b8414
            Reviewed-on: http://gerrit.cloudera.org:8080/5387
            Reviewed-by: Jim Apple <jbapple-impala@apache.org>
            Tested-by: Impala Public Jenkins
        
        Show
        mikesbrown Michael Brown added a comment - commit 54665120cbf4f242688659941ac50bacc8c221c8 Author: Michael Brown <mikeb@cloudera.com> Date: Thu Dec 1 09:27:43 2016 -0800 IMPALA-4355: random query generator: modify statement execution flow to support DML - Rework the discrepancy searcher to run DML statements. We do this by using the query profile to choose a table, copy that table, and generate a statement that will INSERT into that copy. We chose a slow copy over other methods because INSERTing into a copy is a more reliable test that prevents table sizes from getting out of hand or time-consuming replay to reproduce a particular statement. - Introduce a statement generator stub. The real generator work is tracked in IMPALA-4351 and IMPALA-4353. Here we simply generate a basic INSERT INTO ... VALUES statement to make sure our general query execution flow is working. - Add query profile stub for DML statements (INSERT-only at this time). Since we'll want INSERT INTO ... SELECT very soon, this inherits from DefaultProfile. Also add building blocks for choosing random statements in the DefaultProfile. - Improve the concept of an "execution mode" and add new modes. Before, we had "RAW", "CREATE_TABLE_AS", and "CREATE_VIEW_AS". The idea here is that some random SELECT queries could be generated as "CREATE TABLE|VIEW AS" at execution time, based on weights in the query profile. First, we remove the use of raw string literals for this, since raw string literals can be error-prone, and introduce a StatementExecutionMode class to contain a namespace for the enumerated statement execution modes. Second, we introduce a couple new execution modes. The first is DML_SETUP: this is a DML statement that needs to be run in both the test and reference databases concurrently. For our purposes, it's the INSERT ... SELECT that copies data from the chosen random table into the table copy. The second is DML_TEST: this is a randomly-generated DML statement. - Switch to using absolute imports in many places. There was a mix of absolute and relative imports happening here, and they were causing problems, especially when comparing data types. In Python, <class 'db_types.Int'> != <class 'tests.comparison.db_types.Int'>. Using from __future__ import absolute_import didn't seem to catch the relative import usage anyway, so I haven't employed that. - Rename some, but not nearly all, names from "query" to "statement". Doing this is a rather large undertaking leading to much larger diffs and testing (IMPALA-4602). - Fix a handful of flake8 warnings. There are a bunch that went unfixed for over- and under-indentation. - Testing o ./discrepancy_searcher.py runs with and without --explain-only, and with --profile default and --profile dmlonly. For tpch_kudu data, it seems sufficient to use a --timeout of about 300. o Leopard run to make sure standard SELECT-only generation still works o Generated random stress queries locally o Generated random data locally Change-Id: Ia4c63a2223185d0e056cc5713796772e5d1b8414 Reviewed-on: http://gerrit.cloudera.org:8080/5387 Reviewed-by: Jim Apple <jbapple-impala@apache.org> Tested-by: Impala Public Jenkins

          People

          • Assignee:
            mikesbrown Michael Brown
            Reporter:
            mikesbrown Michael Brown
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development