[BEAM-12044] JdbcIO should explicitly setAutoCommit to false

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.28.0
• Fix Version/s: 2.29.0
• Component/s: sdk-java-core
• Assignee: Unassigned
• Reporter: Sylvain Veyrié
• Created: 24/Mar/21 09:41
• Updated: 30/Mar/21 18:54
• Resolved: 27/Mar/21 17:13

Description

Hello,

Per PostgreSQL JDBC documentation, autocommit must be explicitly disabled on the connection to allow cursor streaming.

jkff mentionned it on the mailing list, however even if there is:

poolableConnectionFactory.setDefaultAutoCommit(false);

in JdbcIO:1555, currently, at least with JDBC driver 42.2.16, any read with JdbcIO will memoize the whole dataset (which leads to OOM), since

connection.getAutoCommit()

returns true in JdbcIO#ReadFn#processElement.

I can provide a PR — the patch is pretty simple (and solves the problem for us in 2.28.0):

if (connection == null) {
        connection = dataSource.getConnection();
}
connection.setAutoCommit(false); // line added

Thanks!

Issue Links

• links to: GitHub Pull Request #14338
• links to: GitHub Pull Request #14349