Sqoop
  1. Sqoop
  2. SQOOP-324

Allow use of Avro Data Files with Hive import

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: hive-integration
    • Labels:
      None

      Description

      Hive imports are currently text only. With SQOOP-207 and HIVE-895 it will be possible to use Avro Data Files for storage, which have advantages over plain text, such as supporting compression and a richer set of datatypes.

        Issue Links

          Activity

          Hide
          Yi Liu added a comment -

          Hi Sai Karthik Ganguru, I'm not working on it, please feel free to take it .

          Show
          Yi Liu added a comment - Hi Sai Karthik Ganguru , I'm not working on it, please feel free to take it .
          Hide
          Sai Karthik Ganguru added a comment - - edited

          Yi Liu Are you still working on this? This fix will be very helpful to Sqoop users. If you have any hindrance I can take it up.

          Show
          Sai Karthik Ganguru added a comment - - edited Yi Liu Are you still working on this? This fix will be very helpful to Sqoop users. If you have any hindrance I can take it up.
          Hide
          Jarek Jarcec Cecho added a comment -

          Thank you Yi Liu! I left couple of notes on the review board. Would you mind also share your thoughts about the concerns raised by Tom White in the first comment?

          Show
          Jarek Jarcec Cecho added a comment - Thank you Yi Liu ! I left couple of notes on the review board. Would you mind also share your thoughts about the concerns raised by Tom White in the first comment?
          Hide
          Yi Liu added a comment -

          Jarek, Just upload it as https://reviews.apache.org/r/16124/

          Show
          Yi Liu added a comment - Jarek, Just upload it as https://reviews.apache.org/r/16124/
          Hide
          Yi Liu added a comment -

          sure, I will upload it. Thanks.

          Show
          Yi Liu added a comment - sure, I will upload it. Thanks.
          Hide
          Jarek Jarcec Cecho added a comment -

          Thank you for taking up this JIRA Yi Liu! Would you mind uploading your patch to the review board?

          Show
          Jarek Jarcec Cecho added a comment - Thank you for taking up this JIRA Yi Liu ! Would you mind uploading your patch to the review board ?
          Hide
          Yi Liu added a comment -

          I also required using Avro Data files with Hive import. So I have made this patch and attach it.

          Show
          Yi Liu added a comment - I also required using Avro Data files with Hive import. So I have made this patch and attach it.
          Hide
          Tom White added a comment -

          The paths that need particular care are

          • JDBC Numeric/Decimal (java.math.BigDecimal) -> Avro String -> Hive STRING
          • JDBC Date/Time/Timestamp (java.sql.Date,Time,Timestamp) -> Avro long -> Hive BIGINT

          Note that as it stands, these are different to the paths for text, which are

          • JDBC Numeric/Decimal (java.math.BigDecimal) -> Hive DOUBLE
          • JDBC Date/Time/Timestamp (java.sql.Date,Time,Timestamp) -> Hive STRING
          Show
          Tom White added a comment - The paths that need particular care are JDBC Numeric/Decimal (java.math.BigDecimal) -> Avro String -> Hive STRING JDBC Date/Time/Timestamp (java.sql.Date,Time,Timestamp) -> Avro long -> Hive BIGINT Note that as it stands, these are different to the paths for text, which are JDBC Numeric/Decimal (java.math.BigDecimal) -> Hive DOUBLE JDBC Date/Time/Timestamp (java.sql.Date,Time,Timestamp) -> Hive STRING

            People

            • Assignee:
              Unassigned
              Reporter:
              Tom White
            • Votes:
              8 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:

                Development