Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-2301

Sqoop2: Document how generic jdbc connector generates sql

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: None
    • Labels:
      None

      Description

      There's a lot more information that can added in the Connectors.rst doc. There is a lot of overlap in terms of how some of the SQL is generated for boundary query and the general queries. Here's some information that can be added:

      1. table name - the name of the table to transfer data from
      2. columns - a list of columns to pull from the db
      3. query - SQL statement for querying the data.
      4. boundary query - SQL statement for defining the boundaries.
      5. Partition column - Used for partitioning.

      table name, columns, and query
      Sqoop2 will generate SQL automatically when table name is provided. table name and columns are intended to be used together. If query is specified, table name should not be specified.

      boundary query
      Sqoop2 will generate SQL for the boundary query if not specified. The boundary query is used to define a floor and ceiling for the input splits. These splits define ranges that will be applied to the partition column. If table name is specified, then the SQL generated will take on the form:

      SELECT MIN(`partition column`), MAX(`partition column`) FROM `table name`

      If query is specified, then the generated SQL will take on the form:

      SELECT MIN(`partition column`), MAX(`partition column`) FROM (SELECT ... FROM ... WHERE 1 = 1) SUBQUERY

      partition column
      The partition column will automatically be the private key of the table if not specified.

      Aliases
      Aliases can be used in custom SQL, but keep in mind how the other generated SQL statements will be used.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              abec Abraham Elmahrek
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: