Avro
  1. Avro
  2. AVRO-1269

AVRO is converting ORACLE,Netezza,Teradata decmials & long integers to Strings.

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.7.1
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      I tried to sqoop ORALCE,NETEZZA,TERADATA tables with AVRO foramt using plain JDBC (without using Cloudera connectors). I can see DECIMAL & NUMERIC data types are being converted to AVRO Strings.

      Oracle – NUMBER & INTEGER data types are being converted to AVRO String format.

      NETEZZA – DECIMAL,NUMERIC data types are converted to AVRO String format.

      Teradata – DECIMAL AND LONG data types are converted to AVRO String format.

      When i tried with map-columns to BigDecimal,BigInteger i can see AVRO does not support BigDecimal error message.

      Thanks,
      Prasad Dasari.

        Activity

        Hide
        E. Sammer added a comment -

        Prasad: I'm pretty sure this is a bug in Sqoop. Specifically, it's Sqoop that decides which Avro type to use based on the database column type. Avro just writes the types it's asked to write, according to the schema.

        Show
        E. Sammer added a comment - Prasad: I'm pretty sure this is a bug in Sqoop. Specifically, it's Sqoop that decides which Avro type to use based on the database column type. Avro just writes the types it's asked to write, according to the schema.
        Hide
        Hari Shreedharan added a comment -

        Prasad: I agree with Eric. You should file a jira here: https://issues.apache.org/jira/browse/SQOOP

        Show
        Hari Shreedharan added a comment - Prasad: I agree with Eric. You should file a jira here: https://issues.apache.org/jira/browse/SQOOP
        Hide
        Doug Cutting added a comment -

        Avro's Reflect API does support BigDecimal and BigInteger. They're written with a schema like,

        {"type":"string", "java-class":"java.math.BigDecimal"}

        . Using such a schema, ReflectDatumWriter will write BigDecimal instances and ReflectDatumReader will reconstruct a BigDecimal. This was added in AVRO-1146. We should update the reflect package documentation (http://avro.apache.org/docs/current/api/java/org/apache/avro/reflect/package-summary.html) to better document this.

        Converting SQL's DECIMAL or NUMERIC to Avro's long or float would be lossy.

        Show
        Doug Cutting added a comment - Avro's Reflect API does support BigDecimal and BigInteger. They're written with a schema like, {"type":"string", "java-class":"java.math.BigDecimal"} . Using such a schema, ReflectDatumWriter will write BigDecimal instances and ReflectDatumReader will reconstruct a BigDecimal. This was added in AVRO-1146 . We should update the reflect package documentation ( http://avro.apache.org/docs/current/api/java/org/apache/avro/reflect/package-summary.html ) to better document this. Converting SQL's DECIMAL or NUMERIC to Avro's long or float would be lossy.
        Hide
        Alexandre Normand added a comment -

        Regarding Doug's last comment, I'm making documentation changes as part of AVRO-1268 since that touches Stringable functionality and this should cover the reflect package doc.

        Show
        Alexandre Normand added a comment - Regarding Doug's last comment, I'm making documentation changes as part of AVRO-1268 since that touches Stringable functionality and this should cover the reflect package doc.
        Hide
        Kshitij Pawar added a comment -

        Hi Hari,

        I want to know(in detail) how to move data between Teradata and Hadoop.

        Show
        Kshitij Pawar added a comment - Hi Hari, I want to know(in detail) how to move data between Teradata and Hadoop.

          People

          • Assignee:
            Unassigned
            Reporter:
            Prasad Dasari
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:

              Development