[SQOOP-2787] MySql import and export fails with 5.1 server and 5.1.17+ drivers

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.6
• Fix Version/s: 1.4.7

Environment

MySql 5.1.x server and 5.1.17+ driver

• Assignee: Sowmya Ramesh
• Reporter: Venkat Ranganathan
• Created: 14/Jan/16 18:19
• Updated: 20/Feb/16 07:46
• Resolved: 20/Feb/16 06:44

Description

As discussed in SQOOP-1400 and SQOOP-1617, there are combinations and versions of MySQL server and driver that cause the import and export to fail when the metadata of the tables are being fetched.

16/01/07 21:48:14 DEBUG manager.SqlManager: No connection paramenters specified. Using regular API for making connection.
16/01/07 21:48:14 DEBUG manager.SqlManager: Using fetchSize for next query: -2147483648
16/01/07 21:48:14 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `t1` AS t LIMIT 1
16/01/07 21:48:14 DEBUG manager.SqlManager: Found column col1 of type 4, 11, 0
16/01/07 21:48:14 ERROR manager.SqlManager: Error reading from database: java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@17980bdc is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@17980bdc is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)
at com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingData(MysqlIO.java:2735)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1899)

One potential solution for imports is to add the fetch-size parameter with a valid value to satisfy some of the scenarios when incompatible server/driver versions are used if upgrading either the server or driver is not feasible

Such a workaround is not possible with exports as export operations do not honor the fetch size option. One potential solution for exports is to explicitly set the fetch size for exports to 0 in MySQLManager so that this issue is addressed for exports. Fetching entire result set into memory and OOM issues will not be a factor as we are dealing only with metadata

Issue Links

• links to: RB link