[GRIFFIN-332] JDBC Connector: Ability to Select Specific Columns Instead of All the Columns

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.6.0
• Component/s: accuracy-batch
• Labels: columns jdbc
• Assignee: Unassigned
• Reporter: Obaidul Karim
• Created: 12/Jul/20 03:26
• Updated: 17/Aug/20 04:45

Description

Background:
Thanks to https://issues.apache.org/jira/browse/GRIFFIN-315, we already have JDBC connector.
However, currently, it is pulling all the columns using`"SELECT * FROM $fullTableName"`.
It will cause some issues for larger JDBC tables -

• memory overhead for spark data frame
• longer execution time
• resource overhear for RDBMS

Proposed Improvement:
So, I propose the feature to allow JDBC connector to able to select only required columns.

Example:
We have a rule `"rule":"src.id = tgt.id and src.country = tgt.country "`. Then we only need two columns `id` and 'country'.
So, in connector we can add additional clause `columns` to select only required columns, like below:

 

{   "name":"src",
   "connector":{      "type":"jdbc",
      "config":{         "database":"mydatabase",
         "tablename":"mytable",
         "columns":"id, country",
         "url":"jdbc:sqlserver://myhost:1433;databaseName=mydatabase",
         "user":"user",
         "password":"password",
         "driver":"com.microsoft.sqlserver.jdbc.SQLServerDriver",
         "where":""
      }
   }
}

We can implement it like this, if there is `columns` clause then use it otherwise use `*` as default.

Issue Links

• is cloned by: GRIFFIN-334 Hive Connector: Ability to Select Specific Columns Instead of All the Columns