Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7174

Do not accept string as scale and precision when reading Avro schema

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: None
    • Labels:
      None

      Description

      I've noticed that the current AvroSerde will happily accept schema that uses string instead of integer for scale and precision, e.g. fragment "precision":"4","scale":"1" from following table:

      CREATE TABLE `avro_dec1`(
        `name` string COMMENT 'from deserializer',
        `value` decimal(4,1) COMMENT 'from deserializer')
      COMMENT 'just drop the schema right into the HQL'
      ROW FORMAT SERDE
        'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
      STORED AS INPUTFORMAT
        'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
      OUTPUTFORMAT
        'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
      TBLPROPERTIES (
        'numFiles'='1',
        'avro.schema.literal'='{\"namespace\":\"com.howdy\",\"name\":\"some_schema\",\"type\":\"record\",\"fields\":[{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"value\",\"type\":{\"type\":\"bytes\",\"logicalType\":\"decimal\",\"precision\":\"4\",\"scale\":\"1\"}}]}'
      );
      

      However the Decimal spec defined in AVRO-1402 requires only integer to be there and hence is allowing only following fragment instead "precision":4,"scale":1 (e.g. no double quotes around numbers).

      As Hive can propagate this incorrect schema to new files and hence creating files with invalid schema, I think that we should alter the behavior and insist on the correct schema.

        Attachments

        1. dec.avro
          0.3 kB
          Jarek Jarcec Cecho
        2. HIVE-7174.patch
          2 kB
          Jarek Jarcec Cecho

          Issue Links

            Activity

              People

              • Assignee:
                jarcec Jarek Jarcec Cecho
                Reporter:
                jarcec Jarek Jarcec Cecho
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: