[AVRO-2592] Avro decimal fails on certain conditions - ByteBuffer.position() is the root cause

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.9.1
• Fix Version/s: 1.9.2
• Component/s: java
• Hadoop Flags: Incompatible change, Reviewed
• Assignee: Ryan Skraba
• Reporter: Werner Daehn
• Created: 12/Oct/19 16:30
• Updated: 31/Jan/20 10:52
• Resolved: 31/Jan/20 09:43

Description

The Decimal Conversion from/to Bytebuffer is using the methods that consider the current position, e.g. remaining().

https://github.com/apache/avro/blob/release-1.9.1/lang/java/avro/src/main/java/org/apache/avro/Conversions.java#L82

At first sight that looks like a good idea. But actually it creates all sorts of problems.

For example this code fails with "Zero Length BigInteger":

 

BigDecimal d = BigDecimal.valueOf(3.1415); BigDecimal d = BigDecimal.valueOf(3.1415);

Decimal decimaltype = LogicalTypes.decimal(7, 4); ByteBuffer buffer = DECIMAL_CONVERTER.toBytes(d, null, decimaltype);

System.out.println(DECIMAL_CONVERTER.fromBytes(buffer, null, decimaltype).toString());

BigDecimal n = DECIMAL_CONVERTER.fromBytes(buffer, null, decimaltype);

System.out.println(n.toString());

 

Reason is obvious. The first call to fromBytes() moves the position from 0 to the last byte. The second invocation reads from the last position, hence zero records.

There are other situations this might cause issues, e.g. a user might create the ByteBuffer via other means and it is normal that the position is after the last byte. Then the serialization would not work either. And many other.

As the ByteBuffer is used to wrap a single BigDecimal, I would suggest to remove all position-aware/setting methods and read/write from position zero.

 

 

Issue Links

• relates to: AVRO-2636 GenericData defaultValueCache caches mutable ByteBuffers
• links to: GitHub Pull Request #729