Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-1278

Allow use of uncommitted isolation for databases that support it as an import option

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.3, 1.4.4
    • Fix Version/s: 1.4.5
    • Component/s: None
    • Labels:
      None
    • Environment:

      All DBs that support uncommitted read transaction isolation

      Description

      Some customers might have ETL workflows that depend on data to be imported while the table is otherwise available for the regular data processing. The current read committed isolation can cause failure because of locking semantics used in the database systems used also.

      So, this is to provide a relaxation of the connection isolation used by the mappers to be read uncommitted using a sqoop option. Note that read uncommitted is not a valid option for Oracle so it is not supported with the database and will be ignored for Oracle

      1. SQOOP-1278.diff
        9 kB
        Venkat Ranganathan
      2. SQOOP-1278-3.diff
        10 kB
        Venkat Ranganathan

        Issue Links

          Activity

          Hide
          venkatnrangan Venkat Ranganathan added a comment -

          Uploading new patch

          Show
          venkatnrangan Venkat Ranganathan added a comment - Uploading new patch
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit d3758915bb8b265ec157cd4bc21a7e17d922458e in branch refs/heads/trunk from Jarek Jarcec Cecho
          [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=d375891 ]

          SQOOP-1278: Allow use of uncommitted isolation for databases that support it as an import option

          (Venkat Ranganathan via Jarek Jarcec Cecho)

          Show
          jira-bot ASF subversion and git services added a comment - Commit d3758915bb8b265ec157cd4bc21a7e17d922458e in branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=d375891 ] SQOOP-1278 : Allow use of uncommitted isolation for databases that support it as an import option (Venkat Ranganathan via Jarek Jarcec Cecho)
          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          Thank you Venkat Ranganathan!

          Show
          jarcec Jarek Jarcec Cecho added a comment - Thank you Venkat Ranganathan !
          Hide
          venkatnrangan Venkat Ranganathan added a comment -

          Thanks Jarek Jarcec Cecho for the review and commit

          Show
          venkatnrangan Venkat Ranganathan added a comment - Thanks Jarek Jarcec Cecho for the review and commit
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop20 #875 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/875/)
          SQOOP-1278: Allow use of uncommitted isolation for databases that support it as an import option (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=d3758915bb8b265ec157cd4bc21a7e17d922458e)

          • src/docs/user/common-args.txt
          • src/java/org/apache/sqoop/mapreduce/db/DBConfiguration.java
          • src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java
          • src/docs/user/import.txt
          • src/test/com/cloudera/sqoop/TestSqoopOptions.java
          • src/java/org/apache/sqoop/SqoopOptions.java
          • src/java/org/apache/sqoop/mapreduce/db/DBInputFormat.java
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop20 #875 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/875/ ) SQOOP-1278 : Allow use of uncommitted isolation for databases that support it as an import option (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=d3758915bb8b265ec157cd4bc21a7e17d922458e ) src/docs/user/common-args.txt src/java/org/apache/sqoop/mapreduce/db/DBConfiguration.java src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java src/docs/user/import.txt src/test/com/cloudera/sqoop/TestSqoopOptions.java src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/mapreduce/db/DBInputFormat.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop23 #1077 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/1077/)
          SQOOP-1278: Allow use of uncommitted isolation for databases that support it as an import option (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=d3758915bb8b265ec157cd4bc21a7e17d922458e)

          • src/java/org/apache/sqoop/mapreduce/db/DBInputFormat.java
          • src/docs/user/import.txt
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • src/test/com/cloudera/sqoop/TestSqoopOptions.java
          • src/docs/user/common-args.txt
          • src/java/org/apache/sqoop/mapreduce/db/DBConfiguration.java
          • src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java
          • src/java/org/apache/sqoop/SqoopOptions.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop23 #1077 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/1077/ ) SQOOP-1278 : Allow use of uncommitted isolation for databases that support it as an import option (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=d3758915bb8b265ec157cd4bc21a7e17d922458e ) src/java/org/apache/sqoop/mapreduce/db/DBInputFormat.java src/docs/user/import.txt src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/test/com/cloudera/sqoop/TestSqoopOptions.java src/docs/user/common-args.txt src/java/org/apache/sqoop/mapreduce/db/DBConfiguration.java src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java src/java/org/apache/sqoop/SqoopOptions.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #840 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/840/)
          SQOOP-1278: Allow use of uncommitted isolation for databases that support it as an import option (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=d3758915bb8b265ec157cd4bc21a7e17d922458e)

          • src/test/com/cloudera/sqoop/TestSqoopOptions.java
          • src/java/org/apache/sqoop/mapreduce/db/DBInputFormat.java
          • src/docs/user/import.txt
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • src/java/org/apache/sqoop/mapreduce/db/DBConfiguration.java
          • src/docs/user/common-args.txt
          • src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java
          • src/java/org/apache/sqoop/SqoopOptions.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #840 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/840/ ) SQOOP-1278 : Allow use of uncommitted isolation for databases that support it as an import option (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=d3758915bb8b265ec157cd4bc21a7e17d922458e ) src/test/com/cloudera/sqoop/TestSqoopOptions.java src/java/org/apache/sqoop/mapreduce/db/DBInputFormat.java src/docs/user/import.txt src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/java/org/apache/sqoop/mapreduce/db/DBConfiguration.java src/docs/user/common-args.txt src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java src/java/org/apache/sqoop/SqoopOptions.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop200 #881 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/881/)
          SQOOP-1278: Allow use of uncommitted isolation for databases that support it as an import option (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=d3758915bb8b265ec157cd4bc21a7e17d922458e)

          • src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java
          • src/docs/user/common-args.txt
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • src/docs/user/import.txt
          • src/java/org/apache/sqoop/mapreduce/db/DBConfiguration.java
          • src/java/org/apache/sqoop/SqoopOptions.java
          • src/test/com/cloudera/sqoop/TestSqoopOptions.java
          • src/java/org/apache/sqoop/mapreduce/db/DBInputFormat.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop200 #881 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/881/ ) SQOOP-1278 : Allow use of uncommitted isolation for databases that support it as an import option (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=d3758915bb8b265ec157cd4bc21a7e17d922458e ) src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java src/docs/user/common-args.txt src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/docs/user/import.txt src/java/org/apache/sqoop/mapreduce/db/DBConfiguration.java src/java/org/apache/sqoop/SqoopOptions.java src/test/com/cloudera/sqoop/TestSqoopOptions.java src/java/org/apache/sqoop/mapreduce/db/DBInputFormat.java
          Hide
          dvohra DeepakVohra added a comment -

          Cloudera Sqoop 1.4.5 is still generating the error SQLite supports only TRANSACTION_SERIALIZABLE and TRANSACTION_READ_UNCOMMITTED. Is the support for uncommitted isolation not added to Cloudera Sqoop 1.4.5? Or Cloudera Sqoop 1.4.5 features are added separately from Apache Sqoop?

          Show
          dvohra DeepakVohra added a comment - Cloudera Sqoop 1.4.5 is still generating the error SQLite supports only TRANSACTION_SERIALIZABLE and TRANSACTION_READ_UNCOMMITTED. Is the support for uncommitted isolation not added to Cloudera Sqoop 1.4.5? Or Cloudera Sqoop 1.4.5 features are added separately from Apache Sqoop?
          Hide
          dvohra DeepakVohra added a comment -

          The issue is indicated as fixed in 1.4.5 but getting the following error with Sqoop 1.4.5.
          java.sql.SQLException: SQLite supports only TRANSACTION_SERIALIZABLE and TRANSACTION_READ_UNCOMMITTED.

          Show
          dvohra DeepakVohra added a comment - The issue is indicated as fixed in 1.4.5 but getting the following error with Sqoop 1.4.5. java.sql.SQLException: SQLite supports only TRANSACTION_SERIALIZABLE and TRANSACTION_READ_UNCOMMITTED.
          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          Are you using the --relaxed-isolation parameter, DeepakVohra?

          Show
          jarcec Jarek Jarcec Cecho added a comment - Are you using the --relaxed-isolation parameter, DeepakVohra ?
          Hide
          dvohra DeepakVohra added a comment -

          Thanks Jarek.

          Yes, even with the --relaxed-isolation parameter.

          Show
          dvohra DeepakVohra added a comment - Thanks Jarek. Yes, even with the --relaxed-isolation parameter.
          Hide
          dvohra DeepakVohra added a comment -

          Hadoop version 1.0.0
          Sqoop version 1.4.5
          tar.gz used sqoop-1.4.5.bin__hadoop-1.0.0.tar.gz
          http://apache.mirror.gtcomm.net/sqoop/1.4.5/

          Show
          dvohra DeepakVohra added a comment - Hadoop version 1.0.0 Sqoop version 1.4.5 tar.gz used sqoop-1.4.5.bin__hadoop-1.0.0.tar.gz http://apache.mirror.gtcomm.net/sqoop/1.4.5/
          Hide
          rwieber_cloudera Ryan Wieber added a comment -

          I'm also still getting the "SQLite supports only TRANSACTION_SERIALIZABLE and TRANSACTION_READ_UNCOMMITTED" error even with --relaxed-isolation.

          sqoop 1.4.5

          Command line:
          sqoop import-all-tables --connect jdbc:sqlite:/tmp/PRIVATE.db --m 1 --warehouse-dir /data/PRIVATE --driver org.sqlite.JDBC --relaxed-isolation

          Show
          rwieber_cloudera Ryan Wieber added a comment - I'm also still getting the "SQLite supports only TRANSACTION_SERIALIZABLE and TRANSACTION_READ_UNCOMMITTED" error even with --relaxed-isolation. sqoop 1.4.5 Command line: sqoop import-all-tables --connect jdbc:sqlite:/tmp/PRIVATE.db --m 1 --warehouse-dir /data/PRIVATE --driver org.sqlite.JDBC --relaxed-isolation

            People

            • Assignee:
              venkatnrangan Venkat Ranganathan
              Reporter:
              venkatnrangan Venkat Ranganathan
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development