Sqoop
  1. Sqoop
  2. SQOOP-1073

Sqoop2: Introduce schema for transferred data

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.99.2
    • Fix Version/s: 1.99.3
    • Component/s: None
    • Labels:
      None

      Description

      Sqoop2 similarly as Sqoop1 is not keeping schema of transferred data. As we are planning to modularize entire framework, I believe that schema should be generated and provided to all moving pieces (like extractor, loader, ...).

      1. bugSQOOP-1073.patch
        91 kB
        Jarek Jarcec Cecho
      2. bugSQOOP-1073.patch
        91 kB
        Jarek Jarcec Cecho

        Issue Links

          Activity

          Hide
          Jarek Jarcec Cecho added a comment -

          Attached referenced patch is the entire implementation. It has 90Kb, so I'll split it up into sub tasks for easier review.

          Show
          Jarek Jarcec Cecho added a comment - Attached referenced patch is the entire implementation. It has 90Kb, so I'll split it up into sub tasks for easier review.
          Hide
          Jarek Jarcec Cecho added a comment -

          For reviewers: For the convenience, I've uploaded entire big patch to review board and also slice it into three more or less independent pieces for easier review. Please do finally commit the patch here on SQOOP-1073, not the pieces in the sub tasks. However please comment whenever you see fit, I'll be more than happy to incorporate comments from all reviews.

          Show
          Jarek Jarcec Cecho added a comment - For reviewers: For the convenience, I've uploaded entire big patch to review board and also slice it into three more or less independent pieces for easier review. Please do finally commit the patch here on SQOOP-1073 , not the pieces in the sub tasks. However please comment whenever you see fit, I'll be more than happy to incorporate comments from all reviews.
          Hide
          Sqoop QA bot added a comment -

          Here are the results of testing the latest attachment
          https://issues.apache.org/jira/secure/attachment/12589674/bugSQOOP-1073.patch against branch sqoop2.

          Overall: +1 all checks pass

          SUCCESS: Clean was successful
          SUCCESS: Patch applied correctly
          SUCCESS: Patch compiled
          SUCCESS: All tests passed

          Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/25/console

          This message is automatically generated.

          Show
          Sqoop QA bot added a comment - Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12589674/bugSQOOP-1073.patch against branch sqoop2. Overall: +1 all checks pass SUCCESS: Clean was successful SUCCESS: Patch applied correctly SUCCESS: Patch compiled SUCCESS: All tests passed Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/25/console This message is automatically generated.
          Hide
          ASF subversion and git services added a comment -

          Commit aa8e1e77945cc4ae07549e70d4d481f974a24724 in branch refs/heads/sqoop2 from Venkat Ranganathan
          [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=aa8e1e7 ]

          SQOOP-1073: Sqoop2: Introduce schema for transferred data

          (Jarek Jarcec Cecho via Venkat Ranganathan)

          Show
          ASF subversion and git services added a comment - Commit aa8e1e77945cc4ae07549e70d4d481f974a24724 in branch refs/heads/sqoop2 from Venkat Ranganathan [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=aa8e1e7 ] SQOOP-1073 : Sqoop2: Introduce schema for transferred data (Jarek Jarcec Cecho via Venkat Ranganathan)
          Hide
          Venkat Ranganathan added a comment -

          Thanks Jarcec for this contribution. I have validated the changes and it all looks good.

          Show
          Venkat Ranganathan added a comment - Thanks Jarcec for this contribution. I have validated the changes and it all looks good.
          Hide
          Jarek Jarcec Cecho added a comment -

          Awesome, thank you Venkat Ranganathan!

          Show
          Jarek Jarcec Cecho added a comment - Awesome, thank you Venkat Ranganathan !
          Hide
          Hudson added a comment -

          Integrated in Sqoop2-hadoop200 #424 (See https://builds.apache.org/job/Sqoop2-hadoop200/424/)
          SQOOP-1073: Sqoop2: Introduce schema for transferred data (Revision aa8e1e77945cc4ae07549e70d4d481f974a24724)

          Result = SUCCESS
          n.r.v : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=aa8e1e77945cc4ae07549e70d4d481f974a24724
          Files :

          • common/src/main/java/org/apache/sqoop/schema/type/AbstractString.java
          • common/src/main/java/org/apache/sqoop/schema/type/AbstractNumber.java
          • client/src/main/java/org/apache/sqoop/client/utils/SubmissionDisplayer.java
          • common/src/main/java/org/apache/sqoop/json/SubmissionBean.java
          • common/src/main/java/org/apache/sqoop/model/MSubmission.java
          • common/src/main/java/org/apache/sqoop/schema/type/FloatingPoint.java
          • common/src/test/java/org/apache/sqoop/json/TestSubmissionBean.java
          • common/src/main/java/org/apache/sqoop/schema/type/DateTime.java
          • common/src/main/java/org/apache/sqoop/schema/type/Time.java
          • client/src/main/resources/client-resource.properties
          • common/src/main/java/org/apache/sqoop/schema/type/Text.java
          • common/src/main/java/org/apache/sqoop/schema/Schema.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportInitializer.java
          • common/src/main/java/org/apache/sqoop/json/SchemaBean.java
          • common/src/main/java/org/apache/sqoop/schema/type/Type.java
          • common/src/main/java/org/apache/sqoop/schema/type/AbstractDateTime.java
          • common/src/main/java/org/apache/sqoop/schema/type/Bit.java
          • common/src/main/java/org/apache/sqoop/schema/type/FixedPoint.java
          • spi/src/main/java/org/apache/sqoop/job/etl/Initializer.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/util/SqlTypesUtils.java
          • common/src/test/java/org/apache/sqoop/json/util/TestSchemaSerialization.java
          • common/pom.xml
          • client/src/main/java/org/apache/sqoop/client/core/Constants.java
          • common/src/main/java/org/apache/sqoop/schema/type/Date.java
          • common/src/main/java/org/apache/sqoop/schema/type/Unsupported.java
          • common/src/main/java/org/apache/sqoop/schema/type/Column.java
          • common/src/main/java/org/apache/sqoop/schema/type/Set.java
          • connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportInitializer.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java
          • common/src/main/java/org/apache/sqoop/schema/type/Map.java
          • common/src/main/java/org/apache/sqoop/schema/type/AbstractComplexType.java
          • common/src/main/java/org/apache/sqoop/json/util/SchemaSerialization.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorError.java
          • common/src/main/java/org/apache/sqoop/schema/type/Array.java
          • core/src/main/java/org/apache/sqoop/framework/JobManager.java
          • common/src/main/java/org/apache/sqoop/schema/type/Binary.java
          • common/src/main/java/org/apache/sqoop/schema/SchemaError.java
          • common/src/main/java/org/apache/sqoop/schema/type/Decimal.java
          • common/src/test/java/org/apache/sqoop/json/TestSchemaBean.java
          • common/src/main/java/org/apache/sqoop/schema/type/Enum.java
          Show
          Hudson added a comment - Integrated in Sqoop2-hadoop200 #424 (See https://builds.apache.org/job/Sqoop2-hadoop200/424/ ) SQOOP-1073 : Sqoop2: Introduce schema for transferred data (Revision aa8e1e77945cc4ae07549e70d4d481f974a24724) Result = SUCCESS n.r.v : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=aa8e1e77945cc4ae07549e70d4d481f974a24724 Files : common/src/main/java/org/apache/sqoop/schema/type/AbstractString.java common/src/main/java/org/apache/sqoop/schema/type/AbstractNumber.java client/src/main/java/org/apache/sqoop/client/utils/SubmissionDisplayer.java common/src/main/java/org/apache/sqoop/json/SubmissionBean.java common/src/main/java/org/apache/sqoop/model/MSubmission.java common/src/main/java/org/apache/sqoop/schema/type/FloatingPoint.java common/src/test/java/org/apache/sqoop/json/TestSubmissionBean.java common/src/main/java/org/apache/sqoop/schema/type/DateTime.java common/src/main/java/org/apache/sqoop/schema/type/Time.java client/src/main/resources/client-resource.properties common/src/main/java/org/apache/sqoop/schema/type/Text.java common/src/main/java/org/apache/sqoop/schema/Schema.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportInitializer.java common/src/main/java/org/apache/sqoop/json/SchemaBean.java common/src/main/java/org/apache/sqoop/schema/type/Type.java common/src/main/java/org/apache/sqoop/schema/type/AbstractDateTime.java common/src/main/java/org/apache/sqoop/schema/type/Bit.java common/src/main/java/org/apache/sqoop/schema/type/FixedPoint.java spi/src/main/java/org/apache/sqoop/job/etl/Initializer.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/util/SqlTypesUtils.java common/src/test/java/org/apache/sqoop/json/util/TestSchemaSerialization.java common/pom.xml client/src/main/java/org/apache/sqoop/client/core/Constants.java common/src/main/java/org/apache/sqoop/schema/type/Date.java common/src/main/java/org/apache/sqoop/schema/type/Unsupported.java common/src/main/java/org/apache/sqoop/schema/type/Column.java common/src/main/java/org/apache/sqoop/schema/type/Set.java connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportInitializer.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java common/src/main/java/org/apache/sqoop/schema/type/Map.java common/src/main/java/org/apache/sqoop/schema/type/AbstractComplexType.java common/src/main/java/org/apache/sqoop/json/util/SchemaSerialization.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorError.java common/src/main/java/org/apache/sqoop/schema/type/Array.java core/src/main/java/org/apache/sqoop/framework/JobManager.java common/src/main/java/org/apache/sqoop/schema/type/Binary.java common/src/main/java/org/apache/sqoop/schema/SchemaError.java common/src/main/java/org/apache/sqoop/schema/type/Decimal.java common/src/test/java/org/apache/sqoop/json/TestSchemaBean.java common/src/main/java/org/apache/sqoop/schema/type/Enum.java
          Hide
          Hudson added a comment -

          Integrated in Sqoop2-hadoop100 #447 (See https://builds.apache.org/job/Sqoop2-hadoop100/447/)
          SQOOP-1073: Sqoop2: Introduce schema for transferred data (Revision aa8e1e77945cc4ae07549e70d4d481f974a24724)

          Result = SUCCESS
          n.r.v : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=aa8e1e77945cc4ae07549e70d4d481f974a24724
          Files :

          • common/src/main/java/org/apache/sqoop/schema/type/DateTime.java
          • common/src/main/java/org/apache/sqoop/schema/type/Map.java
          • common/src/main/java/org/apache/sqoop/schema/type/Binary.java
          • common/src/main/java/org/apache/sqoop/schema/type/Array.java
          • core/src/main/java/org/apache/sqoop/framework/JobManager.java
          • common/src/test/java/org/apache/sqoop/json/TestSubmissionBean.java
          • common/src/main/java/org/apache/sqoop/json/util/SchemaSerialization.java
          • common/src/main/java/org/apache/sqoop/schema/type/Date.java
          • spi/src/main/java/org/apache/sqoop/job/etl/Initializer.java
          • common/src/main/java/org/apache/sqoop/schema/type/Text.java
          • common/src/main/java/org/apache/sqoop/json/SubmissionBean.java
          • common/src/main/java/org/apache/sqoop/json/SchemaBean.java
          • common/src/main/java/org/apache/sqoop/schema/type/Column.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorError.java
          • common/src/main/java/org/apache/sqoop/schema/type/Decimal.java
          • common/src/test/java/org/apache/sqoop/json/TestSchemaBean.java
          • client/src/main/resources/client-resource.properties
          • client/src/main/java/org/apache/sqoop/client/utils/SubmissionDisplayer.java
          • common/src/main/java/org/apache/sqoop/schema/type/Set.java
          • common/src/main/java/org/apache/sqoop/schema/type/AbstractDateTime.java
          • common/src/main/java/org/apache/sqoop/model/MSubmission.java
          • common/pom.xml
          • common/src/main/java/org/apache/sqoop/schema/type/Type.java
          • common/src/main/java/org/apache/sqoop/schema/SchemaError.java
          • common/src/main/java/org/apache/sqoop/schema/type/FixedPoint.java
          • connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportInitializer.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/util/SqlTypesUtils.java
          • common/src/main/java/org/apache/sqoop/schema/type/AbstractString.java
          • common/src/main/java/org/apache/sqoop/schema/type/Time.java
          • common/src/main/java/org/apache/sqoop/schema/type/AbstractNumber.java
          • client/src/main/java/org/apache/sqoop/client/core/Constants.java
          • common/src/test/java/org/apache/sqoop/json/util/TestSchemaSerialization.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java
          • common/src/main/java/org/apache/sqoop/schema/type/Bit.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportInitializer.java
          • common/src/main/java/org/apache/sqoop/schema/type/Unsupported.java
          • common/src/main/java/org/apache/sqoop/schema/type/AbstractComplexType.java
          • common/src/main/java/org/apache/sqoop/schema/Schema.java
          • common/src/main/java/org/apache/sqoop/schema/type/Enum.java
          • common/src/main/java/org/apache/sqoop/schema/type/FloatingPoint.java
          Show
          Hudson added a comment - Integrated in Sqoop2-hadoop100 #447 (See https://builds.apache.org/job/Sqoop2-hadoop100/447/ ) SQOOP-1073 : Sqoop2: Introduce schema for transferred data (Revision aa8e1e77945cc4ae07549e70d4d481f974a24724) Result = SUCCESS n.r.v : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=aa8e1e77945cc4ae07549e70d4d481f974a24724 Files : common/src/main/java/org/apache/sqoop/schema/type/DateTime.java common/src/main/java/org/apache/sqoop/schema/type/Map.java common/src/main/java/org/apache/sqoop/schema/type/Binary.java common/src/main/java/org/apache/sqoop/schema/type/Array.java core/src/main/java/org/apache/sqoop/framework/JobManager.java common/src/test/java/org/apache/sqoop/json/TestSubmissionBean.java common/src/main/java/org/apache/sqoop/json/util/SchemaSerialization.java common/src/main/java/org/apache/sqoop/schema/type/Date.java spi/src/main/java/org/apache/sqoop/job/etl/Initializer.java common/src/main/java/org/apache/sqoop/schema/type/Text.java common/src/main/java/org/apache/sqoop/json/SubmissionBean.java common/src/main/java/org/apache/sqoop/json/SchemaBean.java common/src/main/java/org/apache/sqoop/schema/type/Column.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorError.java common/src/main/java/org/apache/sqoop/schema/type/Decimal.java common/src/test/java/org/apache/sqoop/json/TestSchemaBean.java client/src/main/resources/client-resource.properties client/src/main/java/org/apache/sqoop/client/utils/SubmissionDisplayer.java common/src/main/java/org/apache/sqoop/schema/type/Set.java common/src/main/java/org/apache/sqoop/schema/type/AbstractDateTime.java common/src/main/java/org/apache/sqoop/model/MSubmission.java common/pom.xml common/src/main/java/org/apache/sqoop/schema/type/Type.java common/src/main/java/org/apache/sqoop/schema/SchemaError.java common/src/main/java/org/apache/sqoop/schema/type/FixedPoint.java connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportInitializer.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/util/SqlTypesUtils.java common/src/main/java/org/apache/sqoop/schema/type/AbstractString.java common/src/main/java/org/apache/sqoop/schema/type/Time.java common/src/main/java/org/apache/sqoop/schema/type/AbstractNumber.java client/src/main/java/org/apache/sqoop/client/core/Constants.java common/src/test/java/org/apache/sqoop/json/util/TestSchemaSerialization.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java common/src/main/java/org/apache/sqoop/schema/type/Bit.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportInitializer.java common/src/main/java/org/apache/sqoop/schema/type/Unsupported.java common/src/main/java/org/apache/sqoop/schema/type/AbstractComplexType.java common/src/main/java/org/apache/sqoop/schema/Schema.java common/src/main/java/org/apache/sqoop/schema/type/Enum.java common/src/main/java/org/apache/sqoop/schema/type/FloatingPoint.java

            People

            • Assignee:
              Jarek Jarcec Cecho
              Reporter:
              Jarek Jarcec Cecho
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development