Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-416

Allow code-generation arguments with Avro imports

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.2
    • Fix Version/s: 1.4.2
    • Component/s: codegen
    • Labels:

      Description

      If you use --as-avrodatafile, sqoop will automatically generate an avro schema and store it in the header of the avro files. You can easily extract and use that auto-generated schema (e.g., with avro-tools getschema, avro-tools compile). It would be a great improvement, if you could customize the name and namespace of the generated record type in this schema with the already existing arguments --package-name and/or --class-name.

      1. SQOOP-416.patch
        1.0 kB
        Kyryl Bilokurov

        Issue Links

          Activity

          Hide
          kyryl Kyryl Bilokurov added a comment -

          The following patch fixes this issue. Should also apply cleanly to the 1.3.0 version with -p6 (file path stripping, as the file was moved in trunk).

          Show
          kyryl Kyryl Bilokurov added a comment - The following patch fixes this issue. Should also apply cleanly to the 1.3.0 version with -p6 (file path stripping, as the file was moved in trunk).
          Hide
          kyryl Kyryl Bilokurov added a comment -

          Patch attached

          Show
          kyryl Kyryl Bilokurov added a comment - Patch attached
          Hide
          kyryl Kyryl Bilokurov added a comment -

          Newer version, whitespaces cleaned.

          Show
          kyryl Kyryl Bilokurov added a comment - Newer version, whitespaces cleaned.
          Hide
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4161/
          -----------------------------------------------------------

          Review request for Sqoop.

          Summary
          -------

          This is pretty trivial patch, it just introduces new variables for the name and namespace in Avro schema and initializes them to the values passed in the options in the manner similar to ClassWriter.

          This addresses bug SQOOP-416.
          https://issues.apache.org/jira/browse/SQOOP-416

          Diffs


          ./src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 1296327

          Diff: https://reviews.apache.org/r/4161/diff

          Testing
          -------

          Import and export of Avro data files from/to SQL table.

          Thanks,

          Kyryl

          Show
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4161/ ----------------------------------------------------------- Review request for Sqoop. Summary ------- This is pretty trivial patch, it just introduces new variables for the name and namespace in Avro schema and initializes them to the values passed in the options in the manner similar to ClassWriter. This addresses bug SQOOP-416 . https://issues.apache.org/jira/browse/SQOOP-416 Diffs ./src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 1296327 Diff: https://reviews.apache.org/r/4161/diff Testing ------- Import and export of Avro data files from/to SQL table. Thanks, Kyryl
          Hide
          kyryl Kyryl Bilokurov added a comment -

          Resolving, patch is attached and review request submitted.

          Show
          kyryl Kyryl Bilokurov added a comment - Resolving, patch is attached and review request submitted.
          Hide
          kyryl Kyryl Bilokurov added a comment -

          Reopening, code review pending.

          Show
          kyryl Kyryl Bilokurov added a comment - Reopening, code review pending.
          Hide
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4161/
          -----------------------------------------------------------

          (Updated 2012-03-05 09:58:09.819401)

          Review request for Sqoop and Bilung Lee.

          Changes
          -------

          Hi Bilung,

          Could you please review this diff.

          Thanks,
          Kyryl

          Summary
          -------

          This is pretty trivial patch, it just introduces new variables for the name and namespace in Avro schema and initializes them to the values passed in the options in the manner similar to ClassWriter.

          This addresses bug SQOOP-416.
          https://issues.apache.org/jira/browse/SQOOP-416

          Diffs


          ./src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 1296327

          Diff: https://reviews.apache.org/r/4161/diff

          Testing
          -------

          Import and export of Avro data files from/to SQL table.

          Thanks,

          Kyryl

          Show
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4161/ ----------------------------------------------------------- (Updated 2012-03-05 09:58:09.819401) Review request for Sqoop and Bilung Lee. Changes ------- Hi Bilung, Could you please review this diff. Thanks, Kyryl Summary ------- This is pretty trivial patch, it just introduces new variables for the name and namespace in Avro schema and initializes them to the values passed in the options in the manner similar to ClassWriter. This addresses bug SQOOP-416 . https://issues.apache.org/jira/browse/SQOOP-416 Diffs ./src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 1296327 Diff: https://reviews.apache.org/r/4161/diff Testing ------- Import and export of Avro data files from/to SQL table. Thanks, Kyryl
          Hide
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4161/#review5618
          -----------------------------------------------------------

          Ship it!

          Thanks for the patch, Kyryl!

          • Bilung

          On 2012-03-05 09:58:09, Kyryl Bilokurov wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4161/

          -----------------------------------------------------------

          (Updated 2012-03-05 09:58:09)

          Review request for Sqoop and Bilung Lee.

          Summary

          -------

          This is pretty trivial patch, it just introduces new variables for the name and namespace in Avro schema and initializes them to the values passed in the options in the manner similar to ClassWriter.

          This addresses bug SQOOP-416.

          https://issues.apache.org/jira/browse/SQOOP-416

          Diffs

          -----

          ./src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 1296327

          Diff: https://reviews.apache.org/r/4161/diff

          Testing

          -------

          Import and export of Avro data files from/to SQL table.

          Thanks,

          Kyryl

          Show
          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4161/#review5618 ----------------------------------------------------------- Ship it! Thanks for the patch, Kyryl! Bilung On 2012-03-05 09:58:09, Kyryl Bilokurov wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4161/ ----------------------------------------------------------- (Updated 2012-03-05 09:58:09) Review request for Sqoop and Bilung Lee. Summary ------- This is pretty trivial patch, it just introduces new variables for the name and namespace in Avro schema and initializes them to the values passed in the options in the manner similar to ClassWriter. This addresses bug SQOOP-416 . https://issues.apache.org/jira/browse/SQOOP-416 Diffs ----- ./src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 1296327 Diff: https://reviews.apache.org/r/4161/diff Testing ------- Import and export of Avro data files from/to SQL table. Thanks, Kyryl
          Hide
          bleeapache Bilung Lee added a comment -

          Thanks, Kyryl! Patch committed.

          Show
          bleeapache Bilung Lee added a comment - Thanks, Kyryl! Patch committed.
          Hide
          hudson Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6 #93 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/93/)
          SQOOP-416 Allow code-generation arguments with Avro imports (Revision 1297199)

          Result = FAILURE
          blee : http://svn.apache.org/viewvc/?view=rev&rev=1297199
          Files :

          • /incubator/sqoop/trunk/src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java
          Show
          hudson Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6 #93 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/93/ ) SQOOP-416 Allow code-generation arguments with Avro imports (Revision 1297199) Result = FAILURE blee : http://svn.apache.org/viewvc/?view=rev&rev=1297199 Files : /incubator/sqoop/trunk/src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java
          Hide
          vishal.kapoor.in@gmail.com Vishal Kapoor added a comment -

          Is there a way I can get this patch, we are using 1.4.3-cdh4.5.0

          Show
          vishal.kapoor.in@gmail.com Vishal Kapoor added a comment - Is there a way I can get this patch, we are using 1.4.3-cdh4.5.0
          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          Hi Vishal Kapoor, it seems that you are using CDH distribution rather then vanila Apache release and hence you should ask such question on CDH specific forums (mailing list, forum). To answer your question, this patch is part of 1.4.2 version and hence it's also included in 1.4.3 that is base for CDH 4.5.0.

          Show
          jarcec Jarek Jarcec Cecho added a comment - Hi Vishal Kapoor , it seems that you are using CDH distribution rather then vanila Apache release and hence you should ask such question on CDH specific forums ( mailing list , forum ). To answer your question, this patch is part of 1.4.2 version and hence it's also included in 1.4.3 that is base for CDH 4.5.0.
          Hide
          vishal.kapoor.in@gmail.com Vishal Kapoor added a comment -

          Thanks Jarek, sorry about not paying attention but for the benifit of others, the schema and the namespace can be specified by -class-name parameter, this can go to the user guide as well...

          Vishal

          Show
          vishal.kapoor.in@gmail.com Vishal Kapoor added a comment - Thanks Jarek, sorry about not paying attention but for the benifit of others, the schema and the namespace can be specified by -class-name parameter, this can go to the user guide as well... Vishal

            People

            • Assignee:
              kyryl Kyryl Bilokurov
              Reporter:
              akatose Michael Haeusler
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development