Sqoop
  1. Sqoop
  2. SQOOP-1121

Sqoop2: Serialize schema and make them available in the MR job.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.99.2
    • Fix Version/s: 1.99.3
    • Component/s: None
    • Labels:
      None
    1. bugSQOOP-1121.patch
      58 kB
      Jarek Jarcec Cecho
    2. bugSQOOP-1121.patch
      57 kB
      Jarek Jarcec Cecho
    3. bugSQOOP-1121.patch
      57 kB
      Jarek Jarcec Cecho

      Issue Links

        Activity

        Hide
        Jarek Jarcec Cecho added a comment -

        I'm taking this on my queue to continue working on the schema related work.

        Show
        Jarek Jarcec Cecho added a comment - I'm taking this on my queue to continue working on the schema related work.
        Hide
        Sqoop QA bot added a comment -

        Here are the results of testing the latest attachment
        https://issues.apache.org/jira/secure/attachment/12590737/bugSQOOP-1121.patch against branch sqoop2.

        Overall: +1 all checks pass

        SUCCESS: Clean was successful
        SUCCESS: Patch applied correctly
        SUCCESS: Patch compiled
        SUCCESS: All tests passed

        Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/42/console

        This message is automatically generated.

        Show
        Sqoop QA bot added a comment - Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12590737/bugSQOOP-1121.patch against branch sqoop2. Overall: +1 all checks pass SUCCESS: Clean was successful SUCCESS: Patch applied correctly SUCCESS: Patch compiled SUCCESS: All tests passed Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/42/console This message is automatically generated.
        Hide
        Sqoop QA bot added a comment -

        Here are the results of testing the latest attachment
        https://issues.apache.org/jira/secure/attachment/12590737/bugSQOOP-1121.patch against branch sqoop2.

        Overall: -1 due to an error

        ERROR: failed to build with patch (exit code 1)
        SUCCESS: Clean was successful
        SUCCESS: Patch applied correctly

        Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/44/console

        This message is automatically generated.

        Show
        Sqoop QA bot added a comment - Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12590737/bugSQOOP-1121.patch against branch sqoop2. Overall: -1 due to an error ERROR: failed to build with patch (exit code 1) SUCCESS: Clean was successful SUCCESS: Patch applied correctly Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/44/console This message is automatically generated.
        Hide
        Sqoop QA bot added a comment -

        Here are the results of testing the latest attachment
        https://issues.apache.org/jira/secure/attachment/12591461/bugSQOOP-1121.patch against branch sqoop2.

        Overall: +1 all checks pass

        SUCCESS: Clean was successful
        SUCCESS: Patch applied correctly
        SUCCESS: Patch compiled
        SUCCESS: All tests passed

        Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/48/console

        This message is automatically generated.

        Show
        Sqoop QA bot added a comment - Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12591461/bugSQOOP-1121.patch against branch sqoop2. Overall: +1 all checks pass SUCCESS: Clean was successful SUCCESS: Patch applied correctly SUCCESS: Patch compiled SUCCESS: All tests passed Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/48/console This message is automatically generated.
        Hide
        Sqoop QA bot added a comment -

        Here are the results of testing the latest attachment
        https://issues.apache.org/jira/secure/attachment/12591662/bugSQOOP-1121.patch against branch sqoop2.

        Overall: +1 all checks pass

        SUCCESS: Clean was successful
        SUCCESS: Patch applied correctly
        SUCCESS: Patch compiled
        SUCCESS: All tests passed

        Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/52/console

        This message is automatically generated.

        Show
        Sqoop QA bot added a comment - Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12591662/bugSQOOP-1121.patch against branch sqoop2. Overall: +1 all checks pass SUCCESS: Clean was successful SUCCESS: Patch applied correctly SUCCESS: Patch compiled SUCCESS: All tests passed Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/52/console This message is automatically generated.
        Hide
        ASF subversion and git services added a comment -

        Commit 1b2441d21de3eddfabb4091dac84b04f74f8e744 in branch refs/heads/sqoop2 from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=1b2441d ]

        SQOOP-1121. Sqoop2: Serialize schema and make them available in the MR job.

        (Jarek Jarcec Cecho via Hari Shreedharan)

        Show
        ASF subversion and git services added a comment - Commit 1b2441d21de3eddfabb4091dac84b04f74f8e744 in branch refs/heads/sqoop2 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=1b2441d ] SQOOP-1121 . Sqoop2: Serialize schema and make them available in the MR job. (Jarek Jarcec Cecho via Hari Shreedharan)
        Hide
        Hari Shreedharan added a comment -

        Patch committed! Thanks Jarcec!

        Show
        Hari Shreedharan added a comment - Patch committed! Thanks Jarcec!
        Hide
        Hudson added a comment -

        Integrated in Sqoop2-hadoop100 #480 (See https://builds.apache.org/job/Sqoop2-hadoop100/480/)
        SQOOP-1121. Sqoop2: Serialize schema and make them available in the MR job. (Revision 1b2441d21de3eddfabb4091dac84b04f74f8e744)

        Result = FAILURE
        hshreedharan : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=1b2441d21de3eddfabb4091dac84b04f74f8e744
        Files :

        • execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java
        • connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java
        • execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java
        • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java
        • common/src/main/java/org/apache/sqoop/job/etl/ExtractorContext.java
        • common/src/main/java/org/apache/sqoop/job/etl/LoaderContext.java
        • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java
        • execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java
        • execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java
        • connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java
        • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java
        • connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java
        • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/ConfigurationUtils.java
        • execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java
        • execution/mapreduce/pom.xml
        • execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestConfigurationUtils.java
        • common/src/main/java/org/apache/sqoop/job/etl/DestroyerContext.java
        • submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java
        • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java
        • common/src/main/java/org/apache/sqoop/job/etl/PartitionerContext.java
        • core/src/main/java/org/apache/sqoop/framework/JobManager.java
        Show
        Hudson added a comment - Integrated in Sqoop2-hadoop100 #480 (See https://builds.apache.org/job/Sqoop2-hadoop100/480/ ) SQOOP-1121 . Sqoop2: Serialize schema and make them available in the MR job. (Revision 1b2441d21de3eddfabb4091dac84b04f74f8e744) Result = FAILURE hshreedharan : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=1b2441d21de3eddfabb4091dac84b04f74f8e744 Files : execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java common/src/main/java/org/apache/sqoop/job/etl/ExtractorContext.java common/src/main/java/org/apache/sqoop/job/etl/LoaderContext.java execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/ConfigurationUtils.java execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java execution/mapreduce/pom.xml execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestConfigurationUtils.java common/src/main/java/org/apache/sqoop/job/etl/DestroyerContext.java submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java common/src/main/java/org/apache/sqoop/job/etl/PartitionerContext.java core/src/main/java/org/apache/sqoop/framework/JobManager.java
        Hide
        Hudson added a comment -

        Integrated in Sqoop2-hadoop200 #458 (See https://builds.apache.org/job/Sqoop2-hadoop200/458/)
        SQOOP-1121. Sqoop2: Serialize schema and make them available in the MR job. (Revision 1b2441d21de3eddfabb4091dac84b04f74f8e744)

        Result = SUCCESS
        hshreedharan : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=1b2441d21de3eddfabb4091dac84b04f74f8e744
        Files :

        • execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java
        • common/src/main/java/org/apache/sqoop/job/etl/DestroyerContext.java
        • common/src/main/java/org/apache/sqoop/job/etl/ExtractorContext.java
        • execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java
        • execution/mapreduce/pom.xml
        • submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java
        • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/ConfigurationUtils.java
        • execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java
        • core/src/main/java/org/apache/sqoop/framework/JobManager.java
        • connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java
        • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java
        • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java
        • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java
        • execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java
        • common/src/main/java/org/apache/sqoop/job/etl/PartitionerContext.java
        • execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestConfigurationUtils.java
        • execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java
        • execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java
        • common/src/main/java/org/apache/sqoop/job/etl/LoaderContext.java
        • connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java
        • connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java
        Show
        Hudson added a comment - Integrated in Sqoop2-hadoop200 #458 (See https://builds.apache.org/job/Sqoop2-hadoop200/458/ ) SQOOP-1121 . Sqoop2: Serialize schema and make them available in the MR job. (Revision 1b2441d21de3eddfabb4091dac84b04f74f8e744) Result = SUCCESS hshreedharan : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=1b2441d21de3eddfabb4091dac84b04f74f8e744 Files : execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java common/src/main/java/org/apache/sqoop/job/etl/DestroyerContext.java common/src/main/java/org/apache/sqoop/job/etl/ExtractorContext.java execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java execution/mapreduce/pom.xml submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/ConfigurationUtils.java execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java core/src/main/java/org/apache/sqoop/framework/JobManager.java connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java common/src/main/java/org/apache/sqoop/job/etl/PartitionerContext.java execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestConfigurationUtils.java execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java common/src/main/java/org/apache/sqoop/job/etl/LoaderContext.java connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java
        Hide
        Jarek Jarcec Cecho added a comment - - edited

        The failure in profile 100 is valid. I've mistakenly used API method that is not present in Hadoop 1.x and did not run the tests in this profile. My sincere apologies for that! I've open SQOOP-1131 to resolve that.

        Show
        Jarek Jarcec Cecho added a comment - - edited The failure in profile 100 is valid. I've mistakenly used API method that is not present in Hadoop 1.x and did not run the tests in this profile. My sincere apologies for that! I've open SQOOP-1131 to resolve that.
        Hide
        Hari Shreedharan added a comment -

        Even I did not run tests in that profile. Sorry about that!

        Show
        Hari Shreedharan added a comment - Even I did not run tests in that profile. Sorry about that!
        Hide
        ASF subversion and git services added a comment -

        Commit 74ebecb7ce61dbeb98f5b6541992afb69883b75d in branch refs/heads/sqoop2 from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=74ebecb ]

        SQOOP-1131. New test added in SQOOP-1121 is using unsupported API in Hadoop 100

        (Jarek Jarcec Cecho via Hari Shreedharan)

        Show
        ASF subversion and git services added a comment - Commit 74ebecb7ce61dbeb98f5b6541992afb69883b75d in branch refs/heads/sqoop2 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=74ebecb ] SQOOP-1131 . New test added in SQOOP-1121 is using unsupported API in Hadoop 100 (Jarek Jarcec Cecho via Hari Shreedharan)
        Hide
        Hudson added a comment -

        Integrated in Sqoop2-hadoop200 #459 (See https://builds.apache.org/job/Sqoop2-hadoop200/459/)
        SQOOP-1131. New test added in SQOOP-1121 is using unsupported API in Hadoop 100 (Revision 74ebecb7ce61dbeb98f5b6541992afb69883b75d)

        Result = FAILURE
        hshreedharan : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=74ebecb7ce61dbeb98f5b6541992afb69883b75d
        Files :

        • execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestConfigurationUtils.java
        Show
        Hudson added a comment - Integrated in Sqoop2-hadoop200 #459 (See https://builds.apache.org/job/Sqoop2-hadoop200/459/ ) SQOOP-1131 . New test added in SQOOP-1121 is using unsupported API in Hadoop 100 (Revision 74ebecb7ce61dbeb98f5b6541992afb69883b75d) Result = FAILURE hshreedharan : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=74ebecb7ce61dbeb98f5b6541992afb69883b75d Files : execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestConfigurationUtils.java
        Hide
        Hudson added a comment -

        Integrated in Sqoop2-hadoop100 #481 (See https://builds.apache.org/job/Sqoop2-hadoop100/481/)
        SQOOP-1131. New test added in SQOOP-1121 is using unsupported API in Hadoop 100 (Revision 74ebecb7ce61dbeb98f5b6541992afb69883b75d)

        Result = SUCCESS
        hshreedharan : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=74ebecb7ce61dbeb98f5b6541992afb69883b75d
        Files :

        • execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestConfigurationUtils.java
        Show
        Hudson added a comment - Integrated in Sqoop2-hadoop100 #481 (See https://builds.apache.org/job/Sqoop2-hadoop100/481/ ) SQOOP-1131 . New test added in SQOOP-1121 is using unsupported API in Hadoop 100 (Revision 74ebecb7ce61dbeb98f5b6541992afb69883b75d) Result = SUCCESS hshreedharan : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=74ebecb7ce61dbeb98f5b6541992afb69883b75d Files : execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestConfigurationUtils.java

          People

          • Assignee:
            Jarek Jarcec Cecho
            Reporter:
            Hari Shreedharan
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development