Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.2
    • Component/s: None
    • Labels:
      None

      Description

      BLOB is supported for text import already.
      Provide further support for Avro import.

      1. SQOOP-465.patch
        3 kB
        Cheolsoo Park

        Issue Links

        There are no Sub-Tasks for this issue.

          Activity

          Hide
          Bilung Lee added a comment -

          Patch is in. Thanks!

          Show
          Bilung Lee added a comment - Patch is in. Thanks!
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6 #104 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/104/)
          SQOOP-465 BLOB support for Avro import (Revision 1307132)

          Result = SUCCESS
          blee :
          Files :

          • /sqoop/trunk/src/java/org/apache/sqoop/manager/ConnManager.java
          • /sqoop/trunk/src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6 #104 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/104/ ) SQOOP-465 BLOB support for Avro import (Revision 1307132) Result = SUCCESS blee : Files : /sqoop/trunk/src/java/org/apache/sqoop/manager/ConnManager.java /sqoop/trunk/src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4496/#review6547
          -----------------------------------------------------------

          Ship it!

          Thanks for the patch, Cheolsoo!

          • Bilung

          On 2012-03-28 02:46:54, Cheolsoo Park wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4496/

          -----------------------------------------------------------

          (Updated 2012-03-28 02:46:54)

          Review request for Sqoop, Arvind Prabhakar and Bilung Lee.

          Summary

          -------

          BLOB is supported for text and sequence import only.

          Provide further support for Avro import.

          This addresses bug SQOOP-465.

          https://issues.apache.org/jira/browse/SQOOP-465

          Diffs

          -----

          ./src/java/org/apache/sqoop/manager/ConnManager.java 1303136

          ./src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java 1303098

          Diff: https://reviews.apache.org/r/4496/diff

          Testing

          -------

          Tested with Oracle XE 10g and MySQL 5.1 using newly added LobAvroImportTest by SQOOP-470: https://reviews.apache.org/r/4527/

          Ran ant test

          Ran ant test -Dthirdparty=true

          Ran ant checkstyle

          Thanks,

          Cheolsoo

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4496/#review6547 ----------------------------------------------------------- Ship it! Thanks for the patch, Cheolsoo! Bilung On 2012-03-28 02:46:54, Cheolsoo Park wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4496/ ----------------------------------------------------------- (Updated 2012-03-28 02:46:54) Review request for Sqoop, Arvind Prabhakar and Bilung Lee. Summary ------- BLOB is supported for text and sequence import only. Provide further support for Avro import. This addresses bug SQOOP-465 . https://issues.apache.org/jira/browse/SQOOP-465 Diffs ----- ./src/java/org/apache/sqoop/manager/ConnManager.java 1303136 ./src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java 1303098 Diff: https://reviews.apache.org/r/4496/diff Testing ------- Tested with Oracle XE 10g and MySQL 5.1 using newly added LobAvroImportTest by SQOOP-470 : https://reviews.apache.org/r/4527/ Ran ant test Ran ant test -Dthirdparty=true Ran ant checkstyle Thanks, Cheolsoo
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4496/
          -----------------------------------------------------------

          (Updated 2012-03-28 02:46:54.876527)

          Review request for Sqoop, Arvind Prabhakar and Bilung Lee.

          Summary
          -------

          BLOB is supported for text and sequence import only.
          Provide further support for Avro import.

          This addresses bug SQOOP-465.
          https://issues.apache.org/jira/browse/SQOOP-465

          Diffs


          ./src/java/org/apache/sqoop/manager/ConnManager.java 1303136
          ./src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java 1303098

          Diff: https://reviews.apache.org/r/4496/diff

          Testing (updated)
          -------

          Tested with Oracle XE 10g and MySQL 5.1 using newly added LobAvroImportTest by SQOOP-470: https://reviews.apache.org/r/4527/

          Ran ant test
          Ran ant test -Dthirdparty=true
          Ran ant checkstyle

          Thanks,

          Cheolsoo

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4496/ ----------------------------------------------------------- (Updated 2012-03-28 02:46:54.876527) Review request for Sqoop, Arvind Prabhakar and Bilung Lee. Summary ------- BLOB is supported for text and sequence import only. Provide further support for Avro import. This addresses bug SQOOP-465 . https://issues.apache.org/jira/browse/SQOOP-465 Diffs ./src/java/org/apache/sqoop/manager/ConnManager.java 1303136 ./src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java 1303098 Diff: https://reviews.apache.org/r/4496/diff Testing (updated) ------- Tested with Oracle XE 10g and MySQL 5.1 using newly added LobAvroImportTest by SQOOP-470 : https://reviews.apache.org/r/4527/ Ran ant test Ran ant test -Dthirdparty=true Ran ant checkstyle Thanks, Cheolsoo
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4496/
          -----------------------------------------------------------

          (Updated 2012-03-28 02:43:22.864342)

          Review request for Sqoop, Arvind Prabhakar and Bilung Lee.

          Changes
          -------

          Map java.sql.Types.LONGVARBINARY to Avro Bytes.
          MySQL driver maps BLOB to LONGVARBINARY, so it has to be mapped to Avro bytes to support Avro blob import in MySQL.

          Summary
          -------

          BLOB is supported for text and sequence import only.
          Provide further support for Avro import.

          This addresses bug SQOOP-465.
          https://issues.apache.org/jira/browse/SQOOP-465

          Diffs (updated)


          ./src/java/org/apache/sqoop/manager/ConnManager.java 1303136
          ./src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java 1303098

          Diff: https://reviews.apache.org/r/4496/diff

          Testing
          -------

          Manually tested: verified that Avro byte files were generated.

          New JUnit test will be added by SQOOP-470.

          Thanks,

          Cheolsoo

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4496/ ----------------------------------------------------------- (Updated 2012-03-28 02:43:22.864342) Review request for Sqoop, Arvind Prabhakar and Bilung Lee. Changes ------- Map java.sql.Types.LONGVARBINARY to Avro Bytes. MySQL driver maps BLOB to LONGVARBINARY, so it has to be mapped to Avro bytes to support Avro blob import in MySQL. Summary ------- BLOB is supported for text and sequence import only. Provide further support for Avro import. This addresses bug SQOOP-465 . https://issues.apache.org/jira/browse/SQOOP-465 Diffs (updated) ./src/java/org/apache/sqoop/manager/ConnManager.java 1303136 ./src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java 1303098 Diff: https://reviews.apache.org/r/4496/diff Testing ------- Manually tested: verified that Avro byte files were generated. New JUnit test will be added by SQOOP-470 . Thanks, Cheolsoo
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4496/
          -----------------------------------------------------------

          (Updated 2012-03-28 01:56:53.483089)

          Review request for Sqoop, Arvind Prabhakar and Bilung Lee.

          Changes
          -------

          Close LargeObjectLoader in the cleanup method of AvroImportMapper. Not closing LargeObjectLoader can cause EOFException when reading a LOB file via LobReader.

          Summary
          -------

          BLOB is supported for text and sequence import only.
          Provide further support for Avro import.

          This addresses bug SQOOP-465.
          https://issues.apache.org/jira/browse/SQOOP-465

          Diffs (updated)


          ./src/java/org/apache/sqoop/manager/ConnManager.java 1303136
          ./src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java 1303098

          Diff: https://reviews.apache.org/r/4496/diff

          Testing
          -------

          Manually tested: verified that Avro byte files were generated.

          New JUnit test will be added by SQOOP-470.

          Thanks,

          Cheolsoo

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4496/ ----------------------------------------------------------- (Updated 2012-03-28 01:56:53.483089) Review request for Sqoop, Arvind Prabhakar and Bilung Lee. Changes ------- Close LargeObjectLoader in the cleanup method of AvroImportMapper. Not closing LargeObjectLoader can cause EOFException when reading a LOB file via LobReader. Summary ------- BLOB is supported for text and sequence import only. Provide further support for Avro import. This addresses bug SQOOP-465 . https://issues.apache.org/jira/browse/SQOOP-465 Diffs (updated) ./src/java/org/apache/sqoop/manager/ConnManager.java 1303136 ./src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java 1303098 Diff: https://reviews.apache.org/r/4496/diff Testing ------- Manually tested: verified that Avro byte files were generated. New JUnit test will be added by SQOOP-470 . Thanks, Cheolsoo
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4496/
          -----------------------------------------------------------

          Review request for Sqoop, Arvind Prabhakar and Bilung Lee.

          Summary
          -------

          BLOB is supported for text and sequence import only.
          Provide further support for Avro import.

          This addresses bug SQOOP-465.
          https://issues.apache.org/jira/browse/SQOOP-465

          Diffs


          ./src/java/org/apache/sqoop/manager/ConnManager.java 1303136
          ./src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java 1303098

          Diff: https://reviews.apache.org/r/4496/diff

          Testing
          -------

          Manually tested: verified that Avro byte files were generated.

          New JUnit test will be added by SQOOP-470.

          Thanks,

          Cheolsoo

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4496/ ----------------------------------------------------------- Review request for Sqoop, Arvind Prabhakar and Bilung Lee. Summary ------- BLOB is supported for text and sequence import only. Provide further support for Avro import. This addresses bug SQOOP-465 . https://issues.apache.org/jira/browse/SQOOP-465 Diffs ./src/java/org/apache/sqoop/manager/ConnManager.java 1303136 ./src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java 1303098 Diff: https://reviews.apache.org/r/4496/diff Testing ------- Manually tested: verified that Avro byte files were generated. New JUnit test will be added by SQOOP-470 . Thanks, Cheolsoo
          Hide
          Cheolsoo Park added a comment -

          Currently, LargeObjectBlob (LOB) is handled in Sqoop as follows:

          1) if LOB is < MAX_INLINE_LOB_LEN (16 MB by default), it is imported as text (or sequence) files just like any other types of data.

          2) if LOB is > MAX_INLINE_LOB_LEN, it is saved in .lob files, and reference files that contain information about these .lob files are created. The content of reference files looks like this:

          lf,<path>,<offset>,<length>

          But if the --as-sqeuncefile option is enabled, Sqoop generates reference files as sequence files while LOB is still saved in .lob files. (The .lob file format specification can be found at https://github.com/cloudera/sqoop/wiki/sip-3.)

          As the first step of blob support for Avro import, I am going to follow the current semantics of --as-sequencefile. That is, if the --as-avrodatafile option is enabled,

          1) if LOB is < MAX_INLINE_LOB_LEN, it will be saved as Avro data files.

          2) if LOB is > MAX_INLINE_LOB_LEN, reference files will be generated as Avro data files while LOB is still saved in .lob files.

          Show
          Cheolsoo Park added a comment - Currently, LargeObjectBlob (LOB) is handled in Sqoop as follows: 1) if LOB is < MAX_INLINE_LOB_LEN (16 MB by default), it is imported as text (or sequence) files just like any other types of data. 2) if LOB is > MAX_INLINE_LOB_LEN, it is saved in .lob files, and reference files that contain information about these .lob files are created. The content of reference files looks like this: lf,<path>,<offset>,<length> But if the --as-sqeuncefile option is enabled, Sqoop generates reference files as sequence files while LOB is still saved in .lob files. (The .lob file format specification can be found at https://github.com/cloudera/sqoop/wiki/sip-3 .) As the first step of blob support for Avro import, I am going to follow the current semantics of --as-sequencefile. That is, if the --as-avrodatafile option is enabled, 1) if LOB is < MAX_INLINE_LOB_LEN, it will be saved as Avro data files. 2) if LOB is > MAX_INLINE_LOB_LEN, reference files will be generated as Avro data files while LOB is still saved in .lob files.

            People

            • Assignee:
              Cheolsoo Park
              Reporter:
              Bilung Lee
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development