Avro
  1. Avro
  2. AVRO-1518

Python client support decimal.Decimal types -> double encoding / decoding

    Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 1.7.8
    • Component/s: None
    • Labels:
      None
    • Tags:
      python

      Description

      Python standard library > 2.4 provides a Decimal type that has much better semantics then standard binary float. Avro library should be able to accept Decimal's and encode them as doubles.
      (https://docs.python.org/2/library/decimal.html)

      I also believe it should, by default, turn Avro double's into Decimal object instead of a float.

      Simple patch allows for encoding a Decimal into an Avro double

      --- io.py	2014-05-23 13:41:14.000000000 -0700
      +++ /Users/sreynolds/Projects/avro-1.7.6 2/src/avro/io.py	2014-05-23 13:44:03.000000000 -0700
      @@ -46,6 +46,11 @@ try:
       except ImportError:
       	import simplejson as json
      
      +try:
      +        from decimal import Decimal
      +except ImportError:
      +        Decimal = float
      +
       #
       # Constants
       #
      @@ -117,7 +122,7 @@ def validate(expected_schema, datum):
                   and LONG_MIN_VALUE <= datum <= LONG_MAX_VALUE)
         elif schema_type in ['float', 'double']:
           return (isinstance(datum, int) or isinstance(datum, long)
      -            or isinstance(datum, float))
      +            or isinstance(datum, float) or instance(datum, Decimal))
         elif schema_type == 'fixed':
           return isinstance(datum, str) and len(datum) == expected_schema.size
         elif schema_type == 'enum':
      

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        14d 28m 1 Doug Cutting 06/Jun/14 22:38
        Doug Cutting made changes -
        Fix Version/s 1.7.8 [ 12327467 ]
        Fix Version/s 1.7.7 [ 12326041 ]
        Hide
        Steven Moy added a comment -

        Encoding is doable. For decoding, is that going to be a runtime option during decoding by expanding API?

        Also, the patch will need to add an unit test to exercise the encode decimal.Decimal in the existing test suite.

        Show
        Steven Moy added a comment - Encoding is doable. For decoding, is that going to be a runtime option during decoding by expanding API? Also, the patch will need to add an unit test to exercise the encode decimal.Decimal in the existing test suite.
        Doug Cutting made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Assignee Scott Reynolds [ scottreynolds ]
        Fix Version/s 1.7.7 [ 12326041 ]
        Hide
        Doug Cutting added a comment -

        Looks reasonable to me. Any Pythonistas have opinions about this?

        Show
        Doug Cutting added a comment - Looks reasonable to me. Any Pythonistas have opinions about this?
        Scott Reynolds made changes -
        Field Original Value New Value
        Description Python standard library > 2.4 provides a Decimal type that has much better semantics then standard binary float. Avro library should be able to accept Decimal's and encode them as doubles.

        I also believe it should, by default, turn Avro double's into Decimal object instead of a float.

        Simple patch allows for encoding a Decimal into an Avro double
        {code}
        --- io.py 2014-05-23 13:41:14.000000000 -0700
        +++ /Users/sreynolds/Projects/avro-1.7.6 2/src/avro/io.py 2014-05-23 13:44:03.000000000 -0700
        @@ -46,6 +46,11 @@ try:
         except ImportError:
          import simplejson as json

        +try:
        + from decimal import Decimal
        +except ImportError:
        + Decimal = float
        +
         #
         # Constants
         #
        @@ -117,7 +122,7 @@ def validate(expected_schema, datum):
                     and LONG_MIN_VALUE <= datum <= LONG_MAX_VALUE)
           elif schema_type in ['float', 'double']:
             return (isinstance(datum, int) or isinstance(datum, long)
        - or isinstance(datum, float))
        + or isinstance(datum, float) or instance(datum, Decimal))
           elif schema_type == 'fixed':
             return isinstance(datum, str) and len(datum) == expected_schema.size
           elif schema_type == 'enum':
        {code}
        Python standard library > 2.4 provides a Decimal type that has much better semantics then standard binary float. Avro library should be able to accept Decimal's and encode them as doubles.
        (https://docs.python.org/2/library/decimal.html)

        I also believe it should, by default, turn Avro double's into Decimal object instead of a float.

        Simple patch allows for encoding a Decimal into an Avro double
        {code}
        --- io.py 2014-05-23 13:41:14.000000000 -0700
        +++ /Users/sreynolds/Projects/avro-1.7.6 2/src/avro/io.py 2014-05-23 13:44:03.000000000 -0700
        @@ -46,6 +46,11 @@ try:
         except ImportError:
          import simplejson as json

        +try:
        + from decimal import Decimal
        +except ImportError:
        + Decimal = float
        +
         #
         # Constants
         #
        @@ -117,7 +122,7 @@ def validate(expected_schema, datum):
                     and LONG_MIN_VALUE <= datum <= LONG_MAX_VALUE)
           elif schema_type in ['float', 'double']:
             return (isinstance(datum, int) or isinstance(datum, long)
        - or isinstance(datum, float))
        + or isinstance(datum, float) or instance(datum, Decimal))
           elif schema_type == 'fixed':
             return isinstance(datum, str) and len(datum) == expected_schema.size
           elif schema_type == 'enum':
        {code}
        Scott Reynolds created issue -

          People

          • Assignee:
            Scott Reynolds
            Reporter:
            Scott Reynolds
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development