Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3740 Query generator support for Kudu
  3. IMPALA-4354

qgen: implement SqlWriter methods for INSERT statements

        Details

        • Type: Sub-task
        • Status: Resolved
        • Priority: Blocker
        • Resolution: Fixed
        • Affects Version/s: Impala 2.8.0
        • Fix Version/s: Impala 2.8.0
        • Component/s: Infrastructure
        • Labels:
          None

          Description

          Once IMPALA-4343 is finished, implement the SqlWriter methods for both Postgres 9.5+ and Impala for INSERT queries. This involves serializing the Pythonic object model representation of a query and turning it into syntactically valid SQL.

          References:

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            mikesbrown Michael Brown added a comment -

            https://gerrit.cloudera.org/#/c/5162/

            Show
            mikesbrown Michael Brown added a comment - https://gerrit.cloudera.org/#/c/5162/
            Hide
            Permalink
            mikesbrown Michael Brown added a comment - 
            commit 8d4f8d8d9317d2f562cbc7e1517906cd7d6cd8e1
Author: Michael Brown <mikeb@cloudera.com>
Date:   Mon Nov 14 13:53:03 2016 -0800

    IMPALA-4343,IMPALA-4354: qgen: model INSERTs; write INSERTs from query model

    This patch adds support to the random query generator infrastructure to
    model and write SQL INSERTs. It does not actually randomly generate
    INSERTs at this time (tracked in IMPALA-4353 and umbrella task
    IMPALA-3740) but does provide necessary building blocks to do so.

    First, it's necessary to model the INSERTs as part of our data model.
    This was done by taking the current notion of a Query and making it a
    SelectQuery. We also then create an abstract Query containing some of
    the more common methods and attributes. We then model an INSERT query,
    INSERT clause, and VALUES clause (IMPALA-4343).

    Second, it's necessary to test the basics of this data model. It made
    sense to go ahead and implement the necessary SqlWriter methods to write
    the SQL for these clauses (IMPALA-4354).

    I could then use this writer with some existing and new tests that take
    a query written into our data model and write the SQL, verifying they're
    correct.

    For INSERT into Kudu tables, the equivalent PostgreSQL queries need to
    use "ON CONFLICT DO NOTHING", so all existing and new query tests verify
    they can be written as PostgreSQL as well.

    Testing:
    - all the query generator tests pass
    - I can run Leopard front_end.py and load older query generator reports,
      browse them, and re-run failed queries
    - I can run Leopard controller.py to actually do a query generator
      run
    - discrepancy_searcher.py --explain-only ran for hundreds of queries.
      There were no problems writing the SELECT queries

    Change-Id: I38e24da78c49e908449b35f0a6276ebe4236ddba
    Reviewed-on: http://gerrit.cloudera.org:8080/5162
    Reviewed-by: Michael Brown <mikeb@cloudera.com>
    Reviewed-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com>
    Tested-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com>
            Show
            mikesbrown Michael Brown added a comment - commit 8d4f8d8d9317d2f562cbc7e1517906cd7d6cd8e1 Author: Michael Brown <mikeb@cloudera.com> Date: Mon Nov 14 13:53:03 2016 -0800 IMPALA-4343,IMPALA-4354: qgen: model INSERTs; write INSERTs from query model This patch adds support to the random query generator infrastructure to model and write SQL INSERTs. It does not actually randomly generate INSERTs at this time (tracked in IMPALA-4353 and umbrella task IMPALA-3740) but does provide necessary building blocks to do so. First, it's necessary to model the INSERTs as part of our data model. This was done by taking the current notion of a Query and making it a SelectQuery. We also then create an abstract Query containing some of the more common methods and attributes. We then model an INSERT query, INSERT clause, and VALUES clause (IMPALA-4343). Second, it's necessary to test the basics of this data model. It made sense to go ahead and implement the necessary SqlWriter methods to write the SQL for these clauses (IMPALA-4354). I could then use this writer with some existing and new tests that take a query written into our data model and write the SQL, verifying they're correct. For INSERT into Kudu tables, the equivalent PostgreSQL queries need to use "ON CONFLICT DO NOTHING", so all existing and new query tests verify they can be written as PostgreSQL as well. Testing: - all the query generator tests pass - I can run Leopard front_end.py and load older query generator reports, browse them, and re-run failed queries - I can run Leopard controller.py to actually do a query generator run - discrepancy_searcher.py --explain-only ran for hundreds of queries. There were no problems writing the SELECT queries Change-Id: I38e24da78c49e908449b35f0a6276ebe4236ddba Reviewed-on: http://gerrit.cloudera.org:8080/5162 Reviewed-by: Michael Brown <mikeb@cloudera.com> Reviewed-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com> Tested-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com>

              People

              • Assignee:
                mikesbrown Michael Brown
                Reporter:
                mikesbrown Michael Brown
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development