Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3740 Query generator support for Kudu
  3. IMPALA-4352

random query generator: add attributes to Table, Column for tracking of primary keys

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0
    • Fix Version/s: Impala 2.8.0
    • Component/s: Infrastructure
    • Labels:
      None

      Description

      The random query generator needs to keep track of primary keys in the Kudu/Impala tables for CRUD queries. This Jira is about reading table definitions and storing the necessary attributes in the Python object model of the tables. Quick thoughts include a Boolean attribute for a Column as to whether or not it's a primary key, and a sequence attribute for a Table to return the PK column list.

        Activity

        Hide
        mikesbrown Michael Brown added a comment -

        While exploring the PostgresQL syntax for CREATE TABLE with primary keys, INSERT ... ON CONFLICT DO NOTHING, and INSERT ... ON CONFLICT DO UPDATE ("UPSERT"), the "UPSERT"s need a constraint, and a constraint for "IGNORE" is optional. See some of the examples here: https://www.postgresql.org/docs/9.5/static/sql-insert.html

        It might be good, then, to store this constraint as part of the Table for use by the PostgresQL writer. The constraint is named at CREATE TABLE time. More info in IMPALA-4338 about how the table is created.

        Show
        mikesbrown Michael Brown added a comment - While exploring the PostgresQL syntax for CREATE TABLE with primary keys, INSERT ... ON CONFLICT DO NOTHING , and INSERT ... ON CONFLICT DO UPDATE ("UPSERT"), the "UPSERT"s need a constraint, and a constraint for "IGNORE" is optional. See some of the examples here: https://www.postgresql.org/docs/9.5/static/sql-insert.html It might be good, then, to store this constraint as part of the Table for use by the PostgresQL writer. The constraint is named at CREATE TABLE time. More info in IMPALA-4338 about how the table is created.
        Hide
        mikesbrown Michael Brown added a comment -

        Nevermind on the above. Tracking the primary keys should be enough.

        Show
        mikesbrown Michael Brown added a comment - Nevermind on the above. Tracking the primary keys should be enough.
        Show
        mikesbrown Michael Brown added a comment - http://gerrit.cloudera.org:8080/4873
        Hide
        mikesbrown Michael Brown added a comment -
        commit ac516670b66b48b54783be6d5b8816ae2273963c
        Author: Michael Brown <mikeb@cloudera.com>
        Date:   Wed Oct 26 15:38:38 2016 -0700
        
            IMPALA-4352: test infra: store Impala/Kudu primary keys in object model
        
            Test infrastructure, including the random query generator and the data
            migrator, needs to know the primary keys of Impala/Kudu tables. This
            test infrastructure keeps Python object models of the tables and
            columns. This patch adds the ability to read from source Impala/Kudu
            tables via SHOW CREATE TABLE and store primary keys as proper
            attributes. The patch also adds tests that ensure the test
            infrastructure is always able to read and store the primary keys. This
            helps find breakages sooner rather than later. For example, if a
            regression to "SHOW CREATE TABLE" or the test infrastructure makes us no
            longer able to parse primary keys, GVO or other CI will find the
            breakage faster than running the query generator.
        
            I also fixed some flake8 issues in files I touched. There were several
            files that had a lot of white space warnings, and I wanted to keep the
            patch from getting too large.
        
            Change-Id: Ib654b6cd0e8c2a172ffb7330497be4d4a751e6e5
            Reviewed-on: http://gerrit.cloudera.org:8080/4873
            Reviewed-by: Michael Brown <mikeb@cloudera.com>
            Reviewed-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com>
            Tested-by: Internal Jenkins
        
        Show
        mikesbrown Michael Brown added a comment - commit ac516670b66b48b54783be6d5b8816ae2273963c Author: Michael Brown <mikeb@cloudera.com> Date: Wed Oct 26 15:38:38 2016 -0700 IMPALA-4352: test infra: store Impala/Kudu primary keys in object model Test infrastructure, including the random query generator and the data migrator, needs to know the primary keys of Impala/Kudu tables. This test infrastructure keeps Python object models of the tables and columns. This patch adds the ability to read from source Impala/Kudu tables via SHOW CREATE TABLE and store primary keys as proper attributes. The patch also adds tests that ensure the test infrastructure is always able to read and store the primary keys. This helps find breakages sooner rather than later. For example, if a regression to "SHOW CREATE TABLE" or the test infrastructure makes us no longer able to parse primary keys, GVO or other CI will find the breakage faster than running the query generator. I also fixed some flake8 issues in files I touched. There were several files that had a lot of white space warnings, and I wanted to keep the patch from getting too large. Change-Id: Ib654b6cd0e8c2a172ffb7330497be4d4a751e6e5 Reviewed-on: http://gerrit.cloudera.org:8080/4873 Reviewed-by: Michael Brown <mikeb@cloudera.com> Reviewed-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com> Tested-by: Internal Jenkins

          People

          • Assignee:
            mikesbrown Michael Brown
            Reporter:
            mikesbrown Michael Brown
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development