Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-2787

MySql import and export fails with 5.1 server and 5.1.17+ drivers

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.6
    • Fix Version/s: 1.4.7
    • Component/s: None
    • Labels:
      None
    • Environment:

      MySql 5.1.x server and 5.1.17+ driver

      Description

      As discussed in SQOOP-1400 and SQOOP-1617, there are combinations and versions of MySQL server and driver that cause the import and export to fail when the metadata of the tables are being fetched.

      16/01/07 21:48:14 DEBUG manager.SqlManager: No connection paramenters specified. Using regular API for making connection.
      16/01/07 21:48:14 DEBUG manager.SqlManager: Using fetchSize for next query: -2147483648
      16/01/07 21:48:14 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `t1` AS t LIMIT 1
      16/01/07 21:48:14 DEBUG manager.SqlManager: Found column col1 of type 4, 11, 0
      16/01/07 21:48:14 ERROR manager.SqlManager: Error reading from database: java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@17980bdc is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
      java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@17980bdc is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
      at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
      at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)
      at com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingData(MysqlIO.java:2735)
      at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1899)

      One potential solution for imports is to add the fetch-size parameter with a valid value to satisfy some of the scenarios when incompatible server/driver versions are used if upgrading either the server or driver is not feasible

      Such a workaround is not possible with exports as export operations do not honor the fetch size option. One potential solution for exports is to explicitly set the fetch size for exports to 0 in MySQLManager so that this issue is addressed for exports. Fetching entire result set into memory and OOM issues will not be a factor as we are dealing only with metadata

        Attachments

        1. SQOOP-2787.V0.patch
          2 kB
          Sowmya Ramesh
        2. SQOOP-2787.V1.patch
          3 kB
          Sowmya Ramesh

          Issue Links

            Activity

              People

              • Assignee:
                sowmyaramesh Sowmya Ramesh
                Reporter:
                venkatnrangan Venkat Ranganathan
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: