Sqoop
  1. Sqoop
  2. SQOOP-362

Allow user to override mapping when doing import to AVRO format

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.1-incubating
    • Component/s: None
    • Labels:
      None

      Description

      We should allow user to override default mapping even for AVRO file format.

      This is follow up issue for issue SQOOP-342.

      1. SQOOP-362.patch
        8 kB
        Jarek Jarcec Cecho

        Activity

        Jarek Jarcec Cecho created issue -
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2883/
        -----------------------------------------------------------

        Review request for Sqoop and Arvind Prabhakar.

        Summary
        -------

        I've finished user mapping by allowing redefine output for avro as well.

        This addresses bug SQOOP-362.
        https://issues.apache.org/jira/browse/SQOOP-362

        Diffs


        /src/java/com/cloudera/sqoop/SqoopOptions.java 1203816
        /src/java/org/apache/sqoop/SqoopOptions.java 1203816
        /src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 1203816
        /src/java/org/apache/sqoop/tool/BaseSqoopTool.java 1203816
        /src/java/org/apache/sqoop/tool/ImportTool.java 1203816
        /src/test/com/cloudera/sqoop/TestAvroImport.java 1203816

        Diff: https://reviews.apache.org/r/2883/diff

        Testing
        -------

        • Added new test for mapping override
        • Passed standard tests
        • Passed check style

        Thanks,

        Jarek

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2883/ ----------------------------------------------------------- Review request for Sqoop and Arvind Prabhakar. Summary ------- I've finished user mapping by allowing redefine output for avro as well. This addresses bug SQOOP-362 . https://issues.apache.org/jira/browse/SQOOP-362 Diffs /src/java/com/cloudera/sqoop/SqoopOptions.java 1203816 /src/java/org/apache/sqoop/SqoopOptions.java 1203816 /src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 1203816 /src/java/org/apache/sqoop/tool/BaseSqoopTool.java 1203816 /src/java/org/apache/sqoop/tool/ImportTool.java 1203816 /src/test/com/cloudera/sqoop/TestAvroImport.java 1203816 Diff: https://reviews.apache.org/r/2883/diff Testing ------- Added new test for mapping override Passed standard tests Passed check style Thanks, Jarek
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2883/#review3442
        -----------------------------------------------------------

        Ship it!

        Overall the changes look good, with one comment below. Please attach the patch to the JIRA when done.

        /src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java
        <https://reviews.apache.org/r/2883/#comment7685>

        This should be INTEGER instead.

        • Arvind

        On 2011-11-19 20:08:03, Jarek Cecho wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2883/

        -----------------------------------------------------------

        (Updated 2011-11-19 20:08:03)

        Review request for Sqoop and Arvind Prabhakar.

        Summary

        -------

        I've finished user mapping by allowing redefine output for avro as well.

        This addresses bug SQOOP-362.

        https://issues.apache.org/jira/browse/SQOOP-362

        Diffs

        -----

        /src/java/com/cloudera/sqoop/SqoopOptions.java 1203816

        /src/java/org/apache/sqoop/SqoopOptions.java 1203816

        /src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 1203816

        /src/java/org/apache/sqoop/tool/BaseSqoopTool.java 1203816

        /src/java/org/apache/sqoop/tool/ImportTool.java 1203816

        /src/test/com/cloudera/sqoop/TestAvroImport.java 1203816

        Diff: https://reviews.apache.org/r/2883/diff

        Testing

        -------

        * Added new test for mapping override

        * Passed standard tests

        * Passed check style

        Thanks,

        Jarek

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2883/#review3442 ----------------------------------------------------------- Ship it! Overall the changes look good, with one comment below. Please attach the patch to the JIRA when done. /src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java < https://reviews.apache.org/r/2883/#comment7685 > This should be INTEGER instead. Arvind On 2011-11-19 20:08:03, Jarek Cecho wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2883/ ----------------------------------------------------------- (Updated 2011-11-19 20:08:03) Review request for Sqoop and Arvind Prabhakar. Summary ------- I've finished user mapping by allowing redefine output for avro as well. This addresses bug SQOOP-362 . https://issues.apache.org/jira/browse/SQOOP-362 Diffs ----- /src/java/com/cloudera/sqoop/SqoopOptions.java 1203816 /src/java/org/apache/sqoop/SqoopOptions.java 1203816 /src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 1203816 /src/java/org/apache/sqoop/tool/BaseSqoopTool.java 1203816 /src/java/org/apache/sqoop/tool/ImportTool.java 1203816 /src/test/com/cloudera/sqoop/TestAvroImport.java 1203816 Diff: https://reviews.apache.org/r/2883/diff Testing ------- * Added new test for mapping override * Passed standard tests * Passed check style Thanks, Jarek
        Jarek Jarcec Cecho made changes -
        Field Original Value New Value
        Attachment SQOOP-362.patch [ 12504818 ]
        Jarek Jarcec Cecho made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-11-22 19:57:50, Arvind Prabhakar wrote:

        > /src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java, line 117

        > <https://reviews.apache.org/r/2883/diff/1/?file=59472#file59472line117>

        >

        > This should be INTEGER instead.

        Thank you for the review sir. Patch has been attached to the JIRA.

        Jarcec

        • Jarek

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2883/#review3442
        -----------------------------------------------------------

        On 2011-11-19 20:08:03, Jarek Cecho wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2883/

        -----------------------------------------------------------

        (Updated 2011-11-19 20:08:03)

        Review request for Sqoop and Arvind Prabhakar.

        Summary

        -------

        I've finished user mapping by allowing redefine output for avro as well.

        This addresses bug SQOOP-362.

        https://issues.apache.org/jira/browse/SQOOP-362

        Diffs

        -----

        /src/java/com/cloudera/sqoop/SqoopOptions.java 1203816

        /src/java/org/apache/sqoop/SqoopOptions.java 1203816

        /src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 1203816

        /src/java/org/apache/sqoop/tool/BaseSqoopTool.java 1203816

        /src/java/org/apache/sqoop/tool/ImportTool.java 1203816

        /src/test/com/cloudera/sqoop/TestAvroImport.java 1203816

        Diff: https://reviews.apache.org/r/2883/diff

        Testing

        -------

        * Added new test for mapping override

        * Passed standard tests

        * Passed check style

        Thanks,

        Jarek

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-11-22 19:57:50, Arvind Prabhakar wrote: > /src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java, line 117 > < https://reviews.apache.org/r/2883/diff/1/?file=59472#file59472line117 > > > This should be INTEGER instead. Thank you for the review sir. Patch has been attached to the JIRA. Jarcec Jarek ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2883/#review3442 ----------------------------------------------------------- On 2011-11-19 20:08:03, Jarek Cecho wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2883/ ----------------------------------------------------------- (Updated 2011-11-19 20:08:03) Review request for Sqoop and Arvind Prabhakar. Summary ------- I've finished user mapping by allowing redefine output for avro as well. This addresses bug SQOOP-362 . https://issues.apache.org/jira/browse/SQOOP-362 Diffs ----- /src/java/com/cloudera/sqoop/SqoopOptions.java 1203816 /src/java/org/apache/sqoop/SqoopOptions.java 1203816 /src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 1203816 /src/java/org/apache/sqoop/tool/BaseSqoopTool.java 1203816 /src/java/org/apache/sqoop/tool/ImportTool.java 1203816 /src/test/com/cloudera/sqoop/TestAvroImport.java 1203816 Diff: https://reviews.apache.org/r/2883/diff Testing ------- * Added new test for mapping override * Passed standard tests * Passed check style Thanks, Jarek
        Hide
        Arvind Prabhakar added a comment -

        Patch committed. Thanks Jarcec!

        Show
        Arvind Prabhakar added a comment - Patch committed. Thanks Jarcec!
        Arvind Prabhakar made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 1.4.1-incubating [ 12318902 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Sqoop-jdk-1.6 #64 (See https://builds.apache.org/job/Sqoop-jdk-1.6/64/)
        SQOOP-362. Allow user to override type mapping when doing Avro import.

        (Jarcec Cecho via Arvind Prabhakar)

        arvind : http://svn.apache.org/viewvc/?view=rev&rev=1205233
        Files :

        • /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/SqoopOptions.java
        • /incubator/sqoop/trunk/src/java/org/apache/sqoop/SqoopOptions.java
        • /incubator/sqoop/trunk/src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java
        • /incubator/sqoop/trunk/src/java/org/apache/sqoop/tool/BaseSqoopTool.java
        • /incubator/sqoop/trunk/src/java/org/apache/sqoop/tool/ImportTool.java
        • /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroImport.java
        Show
        Hudson added a comment - Integrated in Sqoop-jdk-1.6 #64 (See https://builds.apache.org/job/Sqoop-jdk-1.6/64/ ) SQOOP-362 . Allow user to override type mapping when doing Avro import. (Jarcec Cecho via Arvind Prabhakar) arvind : http://svn.apache.org/viewvc/?view=rev&rev=1205233 Files : /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/SqoopOptions.java /incubator/sqoop/trunk/src/java/org/apache/sqoop/SqoopOptions.java /incubator/sqoop/trunk/src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java /incubator/sqoop/trunk/src/java/org/apache/sqoop/tool/BaseSqoopTool.java /incubator/sqoop/trunk/src/java/org/apache/sqoop/tool/ImportTool.java /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/TestAvroImport.java

          People

          • Assignee:
            Jarek Jarcec Cecho
            Reporter:
            Jarek Jarcec Cecho
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development