Sqoop
  1. Sqoop
  2. SQOOP-843

Sqoop2: Generic JDBC connector is committing transaction on export with autoCommit on

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.99.1
    • Fix Version/s: 1.99.2
    • Component/s: None
    • Labels:
      None

      Description

      I've noticed that Generic JDBC Connector is not explicitly setting autoCommit and yet is committing transactions. This might lead to issues with for example MySQL JDBC driver that sets the autoCommit to true by default and calling commit will result in following exception being thrown:

      2013-01-23 09:05:02,316 ERROR org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor: Error while loading data out of MR job.
      org.apache.sqoop.common.SqoopException: GENERIC_JDBC_CONNECTOR_0002:Unable to execute the SQL statement
      	at org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeBatch(GenericJdbcExecutor.java:106)
      	at org.apache.sqoop.connector.jdbc.GenericJdbcExportLoader.load(GenericJdbcExportLoader.java:66)
      	at org.apache.sqoop.connector.jdbc.GenericJdbcExportLoader.load(GenericJdbcExportLoader.java:26)
      	at org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:212)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      Caused by: java.sql.SQLException: Can't call commit when autocommit=true
      	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:926)
      	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:923)
      	at com.mysql.jdbc.ConnectionImpl.commit(ConnectionImpl.java:1683)
      	at org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeBatch(GenericJdbcExecutor.java:103)
      	... 9 more
      
      1. bugSQOOP-843.patch
        2 kB
        Jarek Jarcec Cecho

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          3m 44s 1 Jarek Jarcec Cecho 23/Jan/13 17:38
          Patch Available Patch Available Resolved Resolved
          7d 5h 46m 1 Cheolsoo Park 30/Jan/13 23:24
          Hide
          Hudson added a comment -

          Integrated in Sqoop2-hadoop100 #157 (See https://builds.apache.org/job/Sqoop2-hadoop100/157/)
          SQOOP-843: Generic JDBC connector is committing transaction on export with autoCommit on (Revision eef227a5f9eee8a4dd04f9016387bbed59c96995)

          Result = SUCCESS
          cheolsoo : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=eef227a5f9eee8a4dd04f9016387bbed59c96995
          Files :

          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportLoader.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExecutor.java
          Show
          Hudson added a comment - Integrated in Sqoop2-hadoop100 #157 (See https://builds.apache.org/job/Sqoop2-hadoop100/157/ ) SQOOP-843 : Generic JDBC connector is committing transaction on export with autoCommit on (Revision eef227a5f9eee8a4dd04f9016387bbed59c96995) Result = SUCCESS cheolsoo : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=eef227a5f9eee8a4dd04f9016387bbed59c96995 Files : connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportLoader.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExecutor.java
          Hide
          Hudson added a comment -

          Integrated in Sqoop2-hadoop200 #171 (See https://builds.apache.org/job/Sqoop2-hadoop200/171/)
          SQOOP-843: Generic JDBC connector is committing transaction on export with autoCommit on (Revision eef227a5f9eee8a4dd04f9016387bbed59c96995)

          Result = SUCCESS
          cheolsoo : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=eef227a5f9eee8a4dd04f9016387bbed59c96995
          Files :

          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExecutor.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportLoader.java
          Show
          Hudson added a comment - Integrated in Sqoop2-hadoop200 #171 (See https://builds.apache.org/job/Sqoop2-hadoop200/171/ ) SQOOP-843 : Generic JDBC connector is committing transaction on export with autoCommit on (Revision eef227a5f9eee8a4dd04f9016387bbed59c96995) Result = SUCCESS cheolsoo : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=eef227a5f9eee8a4dd04f9016387bbed59c96995 Files : connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExecutor.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportLoader.java
          Cheolsoo Park made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Cheolsoo Park added a comment -

          Committed to sqoop2. Thanks Jarcec!

          Show
          Cheolsoo Park added a comment - Committed to sqoop2. Thanks Jarcec!
          Hide
          Cheolsoo Park added a comment -

          +1.

          Show
          Cheolsoo Park added a comment - +1.
          Jarek Jarcec Cecho made changes -
          Attachment bugSQOOP-843.patch [ 12566152 ]
          Jarek Jarcec Cecho made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jarek Jarcec Cecho made changes -
          Field Original Value New Value
          Remote Link This issue links to "Review board (Web Link)" [ 11981 ]
          Jarek Jarcec Cecho created issue -

            People

            • Assignee:
              Jarek Jarcec Cecho
              Reporter:
              Jarek Jarcec Cecho
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development