[IMPALA-4939] DECIMAL multiply overflows early

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.9.0
• Fix Version/s: Impala 2.11.0
• Component/s: Backend
• Epic Link: Numeric Datatype Conversions
• Target Version: Product Backlog
• Assignee: Taras Bobrovytsky
• Reporter: Daniel Hecht
• Created: 15/Feb/17 22:14
• Updated: 21/Nov/18 23:30
• Resolved: 05/Oct/17 06:07

Description

Decimal multiplication indicates overflow before you've actually overflowed the result type. For example:

[dhecht-desktop.ca.cloudera.com:21000] > select typeof(cast(.1 as decimal(20,20)) * cast(1 as decimal(20,19)));
Query: select typeof(cast(.1 as decimal(20,20)) * cast(1 as decimal(20,19)))
Query submitted at: 2017-02-15 14:08:11 (Coordinator: http://dhecht-desktop.ca.cloudera.com:25000)
Query progress can be monitored at: http://dhecht-desktop.ca.cloudera.com:25000/query_plan?query_id=ac4e45202a0850c4:c757c44600000000
+-----------------------------------------------------------------+
| typeof(cast(0.1 as decimal(20,20)) * cast(1 as decimal(20,19))) |
+-----------------------------------------------------------------+
| DECIMAL(38,38)                                                  |
+-----------------------------------------------------------------+
[dhecht-desktop.ca.cloudera.com:21000] > select cast(.1 as decimal(20,20)) * cast(1 as decimal(20,19));
Query: select cast(.1 as decimal(20,20)) * cast(1 as decimal(20,19))
Query submitted at: 2017-02-15 14:07:58 (Coordinator: http://dhecht-desktop.ca.cloudera.com:25000)
Query progress can be monitored at: http://dhecht-desktop.ca.cloudera.com:25000/query_plan?query_id=c94daa3dc258be9c:559353e800000000
+---------------------------------------------------------+
| cast(0.1 as decimal(20,20)) * cast(1 as decimal(20,19)) |
+---------------------------------------------------------+
| NULL                                                    |
+---------------------------------------------------------+
WARNINGS: UDF WARNING: Expression overflowed, returning NULL
[dhecht-desktop.ca.cloudera.com:21000] > select cast(.1 as decimal(20,20)) * cast(1 as decimal(19,18));
Query: select cast(.1 as decimal(20,20)) * cast(1 as decimal(19,18))
Query submitted at: 2017-02-15 14:12:48 (Coordinator: http://dhecht-desktop.ca.cloudera.com:25000)
Query progress can be monitored at: http://dhecht-desktop.ca.cloudera.com:25000/query_plan?query_id=8d4a0189a544e194:12148c3100000000
+---------------------------------------------------------+
| cast(0.1 as decimal(20,20)) * cast(1 as decimal(19,18)) |
+---------------------------------------------------------+
| 0.10000000000000000000000000000000000000                |
+---------------------------------------------------------+
Fetched 1 row(s) in 0.01s

I.e. 0.1 * 1 fits in a DECIMAL(38,38), but our implementation of decimal multiply can overflow when S1+S2 > 38 even if the result would fit the result type.

The problem is that when starting with Decimal16Values, we do a 128 * 128 => 128-bit multiply, rather than 128 * 128 => 256 bit multiply. We check the intermediate result which has full (S1+S2) scale against MAX_UNSCALED_DECIMAL16, even though we later have code to scale down the result as needed to fit the result type.

Issue Links

• blocks: IMPALA-4924 Make DECIMAL_V2 the default mode
• blocks: IMPALA-4940 DECIMAL V2 multiply result type
• is related to: IMPALA-4964 Decimal modulo operator is overflowing