Uploaded image for project: 'Sqoop (Retired)'
  1. Sqoop (Retired)
  2. SQOOP-2471

Support arrays and structs datatypes with Sqoop Hcatalog integration

Details

    • New Feature
    • Status: Resolved
    • Critical
    • Resolution: Duplicate
    • 1.4.6
    • None
    • hive-integration
    • None

    Description

      Currently sqoop import is not able to handle any complex type. On the other side the hive already has support for the following complex types:

      • arrays: ARRAY<data_type>
      • structs: STRUCT<col_name : data_type [COMMENT col_comment], ...>

      Since it is probably not possible to obtain all necessary information about those types from general JDBC database, this feature should somehow use an external information provided by arguments --map-column-java and --map-column-hive.

      For example it could look like this:
      --map-column-java item='inventory_item(name text, supplier_id integer,price numeric)'
      --map-column-hive item='STRUCT<name : string, supplier_id : int, price : decimal>'

      In case no additional information is provided some more general type should be created if possible.

      It should be possible to serialize the complex datatypes values into strings when the Hive target column's type is explicitly set to 'STRING'.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              pbenes Pavel Benes
              Votes:
              2 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Slack

                  Issue deployment