Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-2349

Transaction isolation level for metadata queries should be mutable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.6
    • Fix Version/s: 1.4.7
    • Component/s: None
    • Labels:
      None

      Description

      Databases like PDW have don't accept READ COMMITTED isolation level:

      com.microsoft.sqlserver.jdbc.SQLServerException: Setting IsolationLevel to ReadCommitted is not supported.
              at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:197)
              at com.microsoft.sqlserver.jdbc.TDSTokenHandler.onEOF(tdsparser.java:246)
              at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:83)
              at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:39)
              at com.microsoft.sqlserver.jdbc.SQLServerConnection$1ConnectionCommand.doExecute(SQLServerConnection.java:1441)
              at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:4575)
              at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1400)
              at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectionCommand(SQLServerConnection.java:1446)
              at com.microsoft.sqlserver.jdbc.SQLServerConnection.logon(SQLServerConnection.java:1907)
              at com.microsoft.sqlserver.jdbc.SQLServerConnection.access$000(SQLServerConnection.java:41)
              at com.microsoft.sqlserver.jdbc.SQLServerConnection$LogonCommand.doExecute(SQLServerConnection.java:1893)
              at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:4575)
              at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1400)
              at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:1045)
              at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:817)
              at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:700)
              at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:842)
              at java.sql.DriverManager.getConnection(DriverManager.java:571)
              at java.sql.DriverManager.getConnection(DriverManager.java:215)
              at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:880)
              at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
              at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:739)
              at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:762)
              at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:270)
              at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:241)
              at org.apache.sqoop.manager.SqlManager.getColumnTypesForQuery(SqlManager.java:234)
              at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:304)
              at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1833)
              at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1645)
              at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
              at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
              at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
              at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
              at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
              at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
              at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
              at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
      15/05/06 10:07:34 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter
              at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1651)
              at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
              at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
              at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
              at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
              at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
              at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
              at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
              at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
      

      Let's make it so that the isolation for metadata queries doesn't set the transaction isolation level or make it mutable.

        Issue Links

          Activity

          Hide
          jnylenllc Jeff added a comment -

          This also impacts non-logging informix DBs. Any attempt to set the isolation level on an informix DB that is is_logging=0 will result in a failure.

          Show
          jnylenllc Jeff added a comment - This also impacts non-logging informix DBs. Any attempt to set the isolation level on an informix DB that is is_logging=0 will result in a failure.
          Hide
          maugli Attila Szabo added a comment -

          Patch available on review board:
          https://reviews.apache.org/r/53423/

          Show
          maugli Attila Szabo added a comment - Patch available on review board: https://reviews.apache.org/r/53423/
          Hide
          maugli Attila Szabo added a comment -

          Hi Abraham Elmahrek,

          My proposed change is available online on the review board.
          However I'd become concerned about just setting the transaction isolation level for the metadata queries in the SqlManager, and not setting in in the "data connectors" (e.g. OracleManager, OracleConnectionFactory, DBInputFormat, SqlServerDBRecordReader). IMHO it would make sense to set that isolation level too, but with a different cmd line argument.

          What do you think? Could you please share your thoughts?

          Thanks,
          Attila Szabo

          Show
          maugli Attila Szabo added a comment - Hi Abraham Elmahrek , My proposed change is available online on the review board. However I'd become concerned about just setting the transaction isolation level for the metadata queries in the SqlManager, and not setting in in the "data connectors" (e.g. OracleManager, OracleConnectionFactory, DBInputFormat, SqlServerDBRecordReader). IMHO it would make sense to set that isolation level too, but with a different cmd line argument. What do you think? Could you please share your thoughts? Thanks, Attila Szabo
          Hide
          abec Abraham Elmahrek added a comment -

          Attila Szabo Patch seems good except for some high level thoughts. Please see my reviewboard comment.

          Show
          abec Abraham Elmahrek added a comment - Attila Szabo Patch seems good except for some high level thoughts. Please see my reviewboard comment.
          Hide
          maugli Attila Szabo added a comment -

          Hi Abraham Elmahrek,

          Many thanks for your review. Naming convention has been cleaned up.

          Thanks,
          Attila Szabo

          Show
          maugli Attila Szabo added a comment - Hi Abraham Elmahrek , Many thanks for your review. Naming convention has been cleaned up. Thanks, Attila Szabo
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit b0b7b4779c13101930f402fe3c7d832ea0a503f2 in sqoop's branch refs/heads/trunk from Attila Szabo
          [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=b0b7b47 ]

          SQOOP-2349: Add command line option for setting
          transaction isolation levels for metadata queries

          Show
          jira-bot ASF subversion and git services added a comment - Commit b0b7b4779c13101930f402fe3c7d832ea0a503f2 in sqoop's branch refs/heads/trunk from Attila Szabo [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=b0b7b47 ] SQOOP-2349 : Add command line option for setting transaction isolation levels for metadata queries
          Hide
          maugli Attila Szabo added a comment -

          Change has been merged on trunk.

          Thank you Abraham Elmahrek for you help and review!

          Attila Szabo

          Show
          maugli Attila Szabo added a comment - Change has been merged on trunk. Thank you Abraham Elmahrek for you help and review! Attila Szabo
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build Sqoop-hadoop100 #1036 (See https://builds.apache.org/job/Sqoop-hadoop100/1036/)
          SQOOP-2349: Add command line option for setting transaction isolation (maugli: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=b0b7b4779c13101930f402fe3c7d832ea0a503f2)

          • (edit) src/java/org/apache/sqoop/SqoopOptions.java
          • (add) src/test/org/apache/sqoop/tool/TestImportTool.java
          • (add) src/java/org/apache/sqoop/tool/JDBCTransactionLevels.java
          • (edit) src/java/org/apache/sqoop/manager/SqlManager.java
          • (edit) src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Sqoop-hadoop100 #1036 (See https://builds.apache.org/job/Sqoop-hadoop100/1036/ ) SQOOP-2349 : Add command line option for setting transaction isolation (maugli: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=b0b7b4779c13101930f402fe3c7d832ea0a503f2 ) (edit) src/java/org/apache/sqoop/SqoopOptions.java (add) src/test/org/apache/sqoop/tool/TestImportTool.java (add) src/java/org/apache/sqoop/tool/JDBCTransactionLevels.java (edit) src/java/org/apache/sqoop/manager/SqlManager.java (edit) src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build Sqoop-hadoop200 #1077 (See https://builds.apache.org/job/Sqoop-hadoop200/1077/)
          SQOOP-2349: Add command line option for setting transaction isolation (maugli: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=b0b7b4779c13101930f402fe3c7d832ea0a503f2)

          • (add) src/test/org/apache/sqoop/tool/TestImportTool.java
          • (add) src/java/org/apache/sqoop/tool/JDBCTransactionLevels.java
          • (edit) src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • (edit) src/java/org/apache/sqoop/SqoopOptions.java
          • (edit) src/java/org/apache/sqoop/manager/SqlManager.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Sqoop-hadoop200 #1077 (See https://builds.apache.org/job/Sqoop-hadoop200/1077/ ) SQOOP-2349 : Add command line option for setting transaction isolation (maugli: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=b0b7b4779c13101930f402fe3c7d832ea0a503f2 ) (add) src/test/org/apache/sqoop/tool/TestImportTool.java (add) src/java/org/apache/sqoop/tool/JDBCTransactionLevels.java (edit) src/java/org/apache/sqoop/tool/BaseSqoopTool.java (edit) src/java/org/apache/sqoop/SqoopOptions.java (edit) src/java/org/apache/sqoop/manager/SqlManager.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build Sqoop-hadoop20 #1071 (See https://builds.apache.org/job/Sqoop-hadoop20/1071/)
          SQOOP-2349: Add command line option for setting transaction isolation (maugli: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=b0b7b4779c13101930f402fe3c7d832ea0a503f2)

          • (edit) src/java/org/apache/sqoop/manager/SqlManager.java
          • (add) src/test/org/apache/sqoop/tool/TestImportTool.java
          • (add) src/java/org/apache/sqoop/tool/JDBCTransactionLevels.java
          • (edit) src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • (edit) src/java/org/apache/sqoop/SqoopOptions.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Sqoop-hadoop20 #1071 (See https://builds.apache.org/job/Sqoop-hadoop20/1071/ ) SQOOP-2349 : Add command line option for setting transaction isolation (maugli: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=b0b7b4779c13101930f402fe3c7d832ea0a503f2 ) (edit) src/java/org/apache/sqoop/manager/SqlManager.java (add) src/test/org/apache/sqoop/tool/TestImportTool.java (add) src/java/org/apache/sqoop/tool/JDBCTransactionLevels.java (edit) src/java/org/apache/sqoop/tool/BaseSqoopTool.java (edit) src/java/org/apache/sqoop/SqoopOptions.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build Sqoop-hadoop23 #1273 (See https://builds.apache.org/job/Sqoop-hadoop23/1273/)
          SQOOP-2349: Add command line option for setting transaction isolation (maugli: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=b0b7b4779c13101930f402fe3c7d832ea0a503f2)

          • (edit) src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • (add) src/test/org/apache/sqoop/tool/TestImportTool.java
          • (edit) src/java/org/apache/sqoop/SqoopOptions.java
          • (add) src/java/org/apache/sqoop/tool/JDBCTransactionLevels.java
          • (edit) src/java/org/apache/sqoop/manager/SqlManager.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Sqoop-hadoop23 #1273 (See https://builds.apache.org/job/Sqoop-hadoop23/1273/ ) SQOOP-2349 : Add command line option for setting transaction isolation (maugli: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=b0b7b4779c13101930f402fe3c7d832ea0a503f2 ) (edit) src/java/org/apache/sqoop/tool/BaseSqoopTool.java (add) src/test/org/apache/sqoop/tool/TestImportTool.java (edit) src/java/org/apache/sqoop/SqoopOptions.java (add) src/java/org/apache/sqoop/tool/JDBCTransactionLevels.java (edit) src/java/org/apache/sqoop/manager/SqlManager.java

            People

            • Assignee:
              maugli Attila Szabo
              Reporter:
              abec Abraham Elmahrek
            • Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development