Sqoop
  1. Sqoop
  2. SQOOP-1073

Sqoop2: Introduce schema for transferred data

        Details

        • Type: Improvement Improvement
        • Status: Resolved
        • Priority: Major Major
        • Resolution: Fixed
        • Affects Version/s: 1.99.2
        • Fix Version/s: 1.99.3
        • Component/s: None
        • Labels:
          None

          Description

          Sqoop2 similarly as Sqoop1 is not keeping schema of transferred data. As we are planning to modularize entire framework, I believe that schema should be generated and provided to all moving pieces (like extractor, loader, ...).

          1. Text File
            bugSQOOP-1073.patch
            91 kB
            Jarek Jarcec Cecho
          2. Text File
            bugSQOOP-1073.patch
            91 kB
            Jarek Jarcec Cecho

            Issue Links

            incorporates

            Bug - A problem which impairs or prevents the functions of the product. SQOOP-515 Sqoop2: Datatype support

            • Major - Major loss of function.
            • Resolved

            Improvement - An improvement or enhancement to an existing feature or task. SQOOP-1074 Sqoop2: Allow mapping between source and target schema

            • Major - Major loss of function.
            • Open
            links to

            Web Link Review board

              Activity

              Ascending order - Click to sort in descending order
              Jarek Jarcec Cecho created issue -
              Jarek Jarcec Cecho made changes -
              Field Original Value New Value
              Link This issue incorporates SQOOP-1074 [ SQOOP-1074 ]
              Jarek Jarcec Cecho made changes -
              Link This issue incorporates SQOOP-515 [ SQOOP-515 ]
              Jarek Jarcec Cecho made changes -
              Fix Version/s 2.0.0 [ 12319272 ]
              Jarek Jarcec Cecho made changes -
              Assignee Jarek Jarcec Cecho [ jarcec ]
              Hide
              Permalink
              Jarek Jarcec Cecho added a comment -

              Attached referenced patch is the entire implementation. It has 90Kb, so I'll split it up into sub tasks for easier review.

              Show
              Jarek Jarcec Cecho added a comment - Attached referenced patch is the entire implementation. It has 90Kb, so I'll split it up into sub tasks for easier review.
              Jarek Jarcec Cecho made changes -
              Attachment bugSQOOP-1073.patch [ 12589667 ]
              Jarek Jarcec Cecho made changes -
              Status Open [ 1 ] Patch Available [ 10002 ]
              Fix Version/s 1.99.3 [ 12324298 ]
              Fix Version/s 2.0.0 [ 12319272 ]
              Jarek Jarcec Cecho made changes -
              Attachment rb12092.patch [ 12589671 ]
              Jarek Jarcec Cecho made changes -
              Attachment bugSQOOP-1073.patch [ 12589667 ]
              Jarek Jarcec Cecho made changes -
              Remote Link This issue links to "Review board (Web Link)" [ 12368 ]
              Hide
              Permalink
              Jarek Jarcec Cecho added a comment -

              For reviewers: For the convenience, I've uploaded entire big patch to review board and also slice it into three more or less independent pieces for easier review. Please do finally commit the patch here on SQOOP-1073, not the pieces in the sub tasks. However please comment whenever you see fit, I'll be more than happy to incorporate comments from all reviews.

              Show
              Jarek Jarcec Cecho added a comment - For reviewers: For the convenience, I've uploaded entire big patch to review board and also slice it into three more or less independent pieces for easier review. Please do finally commit the patch here on SQOOP-1073 , not the pieces in the sub tasks. However please comment whenever you see fit, I'll be more than happy to incorporate comments from all reviews.
              Jarek Jarcec Cecho made changes -
              Attachment bugSQOOP-1073.patch [ 12589674 ]
              Jarek Jarcec Cecho made changes -
              Attachment rb12092.patch [ 12589671 ]
              Hide
              Permalink
              Sqoop QA bot added a comment -

              Here are the results of testing the latest attachment
              https://issues.apache.org/jira/secure/attachment/12589674/bugSQOOP-1073.patch against branch sqoop2.

              Overall: +1 all checks pass

              SUCCESS: Clean was successful
              SUCCESS: Patch applied correctly
              SUCCESS: Patch compiled
              SUCCESS: All tests passed

              Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/25/console

              This message is automatically generated.

              Show
              Sqoop QA bot added a comment - Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12589674/bugSQOOP-1073.patch against branch sqoop2. Overall: +1 all checks pass SUCCESS: Clean was successful SUCCESS: Patch applied correctly SUCCESS: Patch compiled SUCCESS: All tests passed Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/25/console This message is automatically generated.
              Jarek Jarcec Cecho made changes -
              Attachment bugSQOOP-1073.patch [ 12590180 ]
              Hide
              Permalink
              ASF subversion and git services added a comment -

              Commit aa8e1e77945cc4ae07549e70d4d481f974a24724 in branch refs/heads/sqoop2 from Venkat Ranganathan
              [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=aa8e1e7 ]

              SQOOP-1073: Sqoop2: Introduce schema for transferred data

              (Jarek Jarcec Cecho via Venkat Ranganathan)

              Show
              ASF subversion and git services added a comment - Commit aa8e1e77945cc4ae07549e70d4d481f974a24724 in branch refs/heads/sqoop2 from Venkat Ranganathan [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=aa8e1e7 ] SQOOP-1073 : Sqoop2: Introduce schema for transferred data (Jarek Jarcec Cecho via Venkat Ranganathan)
              Hide
              Permalink
              Venkat Ranganathan added a comment -

              Thanks Jarcec for this contribution. I have validated the changes and it all looks good.

              Show
              Venkat Ranganathan added a comment - Thanks Jarcec for this contribution. I have validated the changes and it all looks good.
              Venkat Ranganathan made changes -
              Status Patch Available [ 10002 ] Resolved [ 5 ]
              Resolution Fixed [ 1 ]
              Hide
              Permalink
              Jarek Jarcec Cecho added a comment -

              Awesome, thank you Venkat Ranganathan!

              Show
              Jarek Jarcec Cecho added a comment - Awesome, thank you Venkat Ranganathan !
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Sqoop2-hadoop200 #424 (See https://builds.apache.org/job/Sqoop2-hadoop200/424/)
              SQOOP-1073: Sqoop2: Introduce schema for transferred data (Revision aa8e1e77945cc4ae07549e70d4d481f974a24724)

              Result = SUCCESS
              n.r.v : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=aa8e1e77945cc4ae07549e70d4d481f974a24724
              Files :

              • common/src/main/java/org/apache/sqoop/schema/type/AbstractString.java
              • common/src/main/java/org/apache/sqoop/schema/type/AbstractNumber.java
              • client/src/main/java/org/apache/sqoop/client/utils/SubmissionDisplayer.java
              • common/src/main/java/org/apache/sqoop/json/SubmissionBean.java
              • common/src/main/java/org/apache/sqoop/model/MSubmission.java
              • common/src/main/java/org/apache/sqoop/schema/type/FloatingPoint.java
              • common/src/test/java/org/apache/sqoop/json/TestSubmissionBean.java
              • common/src/main/java/org/apache/sqoop/schema/type/DateTime.java
              • common/src/main/java/org/apache/sqoop/schema/type/Time.java
              • client/src/main/resources/client-resource.properties
              • common/src/main/java/org/apache/sqoop/schema/type/Text.java
              • common/src/main/java/org/apache/sqoop/schema/Schema.java
              • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportInitializer.java
              • common/src/main/java/org/apache/sqoop/json/SchemaBean.java
              • common/src/main/java/org/apache/sqoop/schema/type/Type.java
              • common/src/main/java/org/apache/sqoop/schema/type/AbstractDateTime.java
              • common/src/main/java/org/apache/sqoop/schema/type/Bit.java
              • common/src/main/java/org/apache/sqoop/schema/type/FixedPoint.java
              • spi/src/main/java/org/apache/sqoop/job/etl/Initializer.java
              • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/util/SqlTypesUtils.java
              • common/src/test/java/org/apache/sqoop/json/util/TestSchemaSerialization.java
              • common/pom.xml
              • client/src/main/java/org/apache/sqoop/client/core/Constants.java
              • common/src/main/java/org/apache/sqoop/schema/type/Date.java
              • common/src/main/java/org/apache/sqoop/schema/type/Unsupported.java
              • common/src/main/java/org/apache/sqoop/schema/type/Column.java
              • common/src/main/java/org/apache/sqoop/schema/type/Set.java
              • connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportInitializer.java
              • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java
              • common/src/main/java/org/apache/sqoop/schema/type/Map.java
              • common/src/main/java/org/apache/sqoop/schema/type/AbstractComplexType.java
              • common/src/main/java/org/apache/sqoop/json/util/SchemaSerialization.java
              • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorError.java
              • common/src/main/java/org/apache/sqoop/schema/type/Array.java
              • core/src/main/java/org/apache/sqoop/framework/JobManager.java
              • common/src/main/java/org/apache/sqoop/schema/type/Binary.java
              • common/src/main/java/org/apache/sqoop/schema/SchemaError.java
              • common/src/main/java/org/apache/sqoop/schema/type/Decimal.java
              • common/src/test/java/org/apache/sqoop/json/TestSchemaBean.java
              • common/src/main/java/org/apache/sqoop/schema/type/Enum.java
              Show
              Hudson added a comment - Integrated in Sqoop2-hadoop200 #424 (See https://builds.apache.org/job/Sqoop2-hadoop200/424/ ) SQOOP-1073 : Sqoop2: Introduce schema for transferred data (Revision aa8e1e77945cc4ae07549e70d4d481f974a24724) Result = SUCCESS n.r.v : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=aa8e1e77945cc4ae07549e70d4d481f974a24724 Files : common/src/main/java/org/apache/sqoop/schema/type/AbstractString.java common/src/main/java/org/apache/sqoop/schema/type/AbstractNumber.java client/src/main/java/org/apache/sqoop/client/utils/SubmissionDisplayer.java common/src/main/java/org/apache/sqoop/json/SubmissionBean.java common/src/main/java/org/apache/sqoop/model/MSubmission.java common/src/main/java/org/apache/sqoop/schema/type/FloatingPoint.java common/src/test/java/org/apache/sqoop/json/TestSubmissionBean.java common/src/main/java/org/apache/sqoop/schema/type/DateTime.java common/src/main/java/org/apache/sqoop/schema/type/Time.java client/src/main/resources/client-resource.properties common/src/main/java/org/apache/sqoop/schema/type/Text.java common/src/main/java/org/apache/sqoop/schema/Schema.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportInitializer.java common/src/main/java/org/apache/sqoop/json/SchemaBean.java common/src/main/java/org/apache/sqoop/schema/type/Type.java common/src/main/java/org/apache/sqoop/schema/type/AbstractDateTime.java common/src/main/java/org/apache/sqoop/schema/type/Bit.java common/src/main/java/org/apache/sqoop/schema/type/FixedPoint.java spi/src/main/java/org/apache/sqoop/job/etl/Initializer.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/util/SqlTypesUtils.java common/src/test/java/org/apache/sqoop/json/util/TestSchemaSerialization.java common/pom.xml client/src/main/java/org/apache/sqoop/client/core/Constants.java common/src/main/java/org/apache/sqoop/schema/type/Date.java common/src/main/java/org/apache/sqoop/schema/type/Unsupported.java common/src/main/java/org/apache/sqoop/schema/type/Column.java common/src/main/java/org/apache/sqoop/schema/type/Set.java connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportInitializer.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java common/src/main/java/org/apache/sqoop/schema/type/Map.java common/src/main/java/org/apache/sqoop/schema/type/AbstractComplexType.java common/src/main/java/org/apache/sqoop/json/util/SchemaSerialization.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorError.java common/src/main/java/org/apache/sqoop/schema/type/Array.java core/src/main/java/org/apache/sqoop/framework/JobManager.java common/src/main/java/org/apache/sqoop/schema/type/Binary.java common/src/main/java/org/apache/sqoop/schema/SchemaError.java common/src/main/java/org/apache/sqoop/schema/type/Decimal.java common/src/test/java/org/apache/sqoop/json/TestSchemaBean.java common/src/main/java/org/apache/sqoop/schema/type/Enum.java
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Sqoop2-hadoop100 #447 (See https://builds.apache.org/job/Sqoop2-hadoop100/447/)
              SQOOP-1073: Sqoop2: Introduce schema for transferred data (Revision aa8e1e77945cc4ae07549e70d4d481f974a24724)

              Result = SUCCESS
              n.r.v : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=aa8e1e77945cc4ae07549e70d4d481f974a24724
              Files :

              • common/src/main/java/org/apache/sqoop/schema/type/DateTime.java
              • common/src/main/java/org/apache/sqoop/schema/type/Map.java
              • common/src/main/java/org/apache/sqoop/schema/type/Binary.java
              • common/src/main/java/org/apache/sqoop/schema/type/Array.java
              • core/src/main/java/org/apache/sqoop/framework/JobManager.java
              • common/src/test/java/org/apache/sqoop/json/TestSubmissionBean.java
              • common/src/main/java/org/apache/sqoop/json/util/SchemaSerialization.java
              • common/src/main/java/org/apache/sqoop/schema/type/Date.java
              • spi/src/main/java/org/apache/sqoop/job/etl/Initializer.java
              • common/src/main/java/org/apache/sqoop/schema/type/Text.java
              • common/src/main/java/org/apache/sqoop/json/SubmissionBean.java
              • common/src/main/java/org/apache/sqoop/json/SchemaBean.java
              • common/src/main/java/org/apache/sqoop/schema/type/Column.java
              • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorError.java
              • common/src/main/java/org/apache/sqoop/schema/type/Decimal.java
              • common/src/test/java/org/apache/sqoop/json/TestSchemaBean.java
              • client/src/main/resources/client-resource.properties
              • client/src/main/java/org/apache/sqoop/client/utils/SubmissionDisplayer.java
              • common/src/main/java/org/apache/sqoop/schema/type/Set.java
              • common/src/main/java/org/apache/sqoop/schema/type/AbstractDateTime.java
              • common/src/main/java/org/apache/sqoop/model/MSubmission.java
              • common/pom.xml
              • common/src/main/java/org/apache/sqoop/schema/type/Type.java
              • common/src/main/java/org/apache/sqoop/schema/SchemaError.java
              • common/src/main/java/org/apache/sqoop/schema/type/FixedPoint.java
              • connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportInitializer.java
              • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/util/SqlTypesUtils.java
              • common/src/main/java/org/apache/sqoop/schema/type/AbstractString.java
              • common/src/main/java/org/apache/sqoop/schema/type/Time.java
              • common/src/main/java/org/apache/sqoop/schema/type/AbstractNumber.java
              • client/src/main/java/org/apache/sqoop/client/core/Constants.java
              • common/src/test/java/org/apache/sqoop/json/util/TestSchemaSerialization.java
              • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java
              • common/src/main/java/org/apache/sqoop/schema/type/Bit.java
              • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportInitializer.java
              • common/src/main/java/org/apache/sqoop/schema/type/Unsupported.java
              • common/src/main/java/org/apache/sqoop/schema/type/AbstractComplexType.java
              • common/src/main/java/org/apache/sqoop/schema/Schema.java
              • common/src/main/java/org/apache/sqoop/schema/type/Enum.java
              • common/src/main/java/org/apache/sqoop/schema/type/FloatingPoint.java
              Show
              Hudson added a comment - Integrated in Sqoop2-hadoop100 #447 (See https://builds.apache.org/job/Sqoop2-hadoop100/447/ ) SQOOP-1073 : Sqoop2: Introduce schema for transferred data (Revision aa8e1e77945cc4ae07549e70d4d481f974a24724) Result = SUCCESS n.r.v : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=aa8e1e77945cc4ae07549e70d4d481f974a24724 Files : common/src/main/java/org/apache/sqoop/schema/type/DateTime.java common/src/main/java/org/apache/sqoop/schema/type/Map.java common/src/main/java/org/apache/sqoop/schema/type/Binary.java common/src/main/java/org/apache/sqoop/schema/type/Array.java core/src/main/java/org/apache/sqoop/framework/JobManager.java common/src/test/java/org/apache/sqoop/json/TestSubmissionBean.java common/src/main/java/org/apache/sqoop/json/util/SchemaSerialization.java common/src/main/java/org/apache/sqoop/schema/type/Date.java spi/src/main/java/org/apache/sqoop/job/etl/Initializer.java common/src/main/java/org/apache/sqoop/schema/type/Text.java common/src/main/java/org/apache/sqoop/json/SubmissionBean.java common/src/main/java/org/apache/sqoop/json/SchemaBean.java common/src/main/java/org/apache/sqoop/schema/type/Column.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorError.java common/src/main/java/org/apache/sqoop/schema/type/Decimal.java common/src/test/java/org/apache/sqoop/json/TestSchemaBean.java client/src/main/resources/client-resource.properties client/src/main/java/org/apache/sqoop/client/utils/SubmissionDisplayer.java common/src/main/java/org/apache/sqoop/schema/type/Set.java common/src/main/java/org/apache/sqoop/schema/type/AbstractDateTime.java common/src/main/java/org/apache/sqoop/model/MSubmission.java common/pom.xml common/src/main/java/org/apache/sqoop/schema/type/Type.java common/src/main/java/org/apache/sqoop/schema/SchemaError.java common/src/main/java/org/apache/sqoop/schema/type/FixedPoint.java connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportInitializer.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/util/SqlTypesUtils.java common/src/main/java/org/apache/sqoop/schema/type/AbstractString.java common/src/main/java/org/apache/sqoop/schema/type/Time.java common/src/main/java/org/apache/sqoop/schema/type/AbstractNumber.java client/src/main/java/org/apache/sqoop/client/core/Constants.java common/src/test/java/org/apache/sqoop/json/util/TestSchemaSerialization.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java common/src/main/java/org/apache/sqoop/schema/type/Bit.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportInitializer.java common/src/main/java/org/apache/sqoop/schema/type/Unsupported.java common/src/main/java/org/apache/sqoop/schema/type/AbstractComplexType.java common/src/main/java/org/apache/sqoop/schema/Schema.java common/src/main/java/org/apache/sqoop/schema/type/Enum.java common/src/main/java/org/apache/sqoop/schema/type/FloatingPoint.java

                People

                • Assignee:
                  Jarek Jarcec Cecho
                  Reporter:
                  Jarek Jarcec Cecho
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  5 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development