[DRILL-6794] Document the JDBC properties required to retrieve result sets in batches while querying large tables

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.14.0
• Component/s: Documentation
• Labels: doc-impacting
• Assignee: Bridget Bevens
• Reporter: Rahul Raj
• Created: 14/Oct/18 03:24
• Updated: 03/Feb/20 17:45

Description

Document the JDBC properties required to retrieve result sets in batches while querying large tables

Querying large tables using JDBC plugin causes OOM as most JDBC drivers cache the entire result set at the client by default.

To avoid this additional parameters needs to be specified with the JDBC connection string so that the driver fetches records in batches and reloads when exhausted.

For postgres driver set autocommit mode to false - jdbc:postgresql://url:port/schema?defaultAutoCommit=false

Links

[1] https://issues.apache.org/jira/browse/DRILL-4177

[2] https://jdbc.postgresql.org/documentation/93/query.html#fetchsize-example

[3] https://www.postgresql.org/docs/9.3/static/ecpg-sql-set-autocommit.html

[4] https://jdbc.postgresql.org/documentation/head/ds-cpds.htm

Issue Links

• relates to: DRILL-4177 select * from table;Node ran out of Heap memory, exiting.java.lang.OutOfMemoryError: GC overhead limit exceeded