[ARROW-2153] [C++/Python] Decimal conversion not working for exponential notation

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.0
• Fix Version/s: 0.9.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/18120
• Assignee: Phillip Cloud
• Reporter: Antony Mayi
• Created: 14/Feb/18 07:30
• Updated: 11/Jan/23 07:19
• Resolved: 01/Mar/18 22:27

Description

import pyarrow as pa
import pandas as pd
import decimal

pa.Table.from_pandas(pd.DataFrame({'a': [decimal.Decimal('1.1'), decimal.Decimal('2E+1')]}))

 

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pyarrow/table.pxi", line 875, in pyarrow.lib.Table.from_pandas (/arrow/python/build/temp.linux-x86_64-3.6/lib.cxx:44927)
  File "/home/skadlec/.local/lib/python3.6/site-packages/pyarrow/pandas_compat.py", line 350, in dataframe_to_arrays
    convert_types)]
  File "/home/skadlec/.local/lib/python3.6/site-packages/pyarrow/pandas_compat.py", line 349, in <listcomp>
    for c, t in zip(columns_to_convert,
  File "/home/skadlec/.local/lib/python3.6/site-packages/pyarrow/pandas_compat.py", line 345, in convert_column
    return pa.array(col, from_pandas=True, type=ty)
  File "pyarrow/array.pxi", line 170, in pyarrow.lib.array (/arrow/python/build/temp.linux-x86_64-3.6/lib.cxx:29224)
  File "pyarrow/array.pxi", line 70, in pyarrow.lib._ndarray_to_array (/arrow/python/build/temp.linux-x86_64-3.6/lib.cxx:28465)
  File "pyarrow/error.pxi", line 77, in pyarrow.lib.check_status (/arrow/python/build/temp.linux-x86_64-3.6/lib.cxx:8270)
pyarrow.lib.ArrowInvalid: Expected base ten digit or decimal point but found 'E' instead.

In manual cases clearly we can write decimal.Decimal('20') instead of decimal.Decimal('2E+1') but during arithmetical operations inside an application the exponential notation can be produced out of control (it is actually the normalized form of the decimal number) plus for some values the exponential notation is the only form expressing the significance so this should be accepted.

The documentation suggests using following transformation but that's only possible when the significance information doesn't need to be kept:

def remove_exponent(d):
    return d.quantize(Decimal(1)) if d == d.to_integral() else d.normalize()

Issue Links

• relates to: ARROW-2145 [Python] Decimal conversion not working for NaN values
• links to: GitHub Pull Request #1618