[IMPALA-2515] Impala rejects Parquet schemas where decimal fixed_len_byte_array columns have unnecessary padding bytes

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: Impala 2.3.0
• Fix Version/s: Impala 4.0.0
• Component/s: Backend
• Labels: ramp-up
• Target Version: Impala 4.0.0
• Assignee: Tim Armstrong
• Reporter: Taras Bobrovytsky
• Created: 09/Oct/15 00:45
• Updated: 18/Jun/20 21:41
• Resolved: 18/Jun/20 21:41

Description

Impala cannot read this:

{"name": "tmp_1",
 "type": "fixed",
 "size": 8,
 "logicalType": "decimal",
 "precision": 10,
 "scale": 5}

However, this can be read:

{"name": "tmp_1",
 "type": "fixed",
 "size": 5,
 "logicalType": "decimal",
 "precision": 10,
 "scale": 5}

Size must be precisely set to this, or Impala is unable to read the decimal column:

size = int(math.ceil((math.log(2, 10) + precision) / math.log(256, 10)))

There is nothing in the Parquet spec that says that Decimal columns must be sized precisely. Arguably it's a bug in the writer if it's doing it, because it's just wasting space.
https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#decimal

Issue Links

• is related to: IMPALA-2494 Impala Unable to scan a Decimal column stored as Bytes
• relates to: IMPALA-5542 Impala cannot scan Parquet decimal stored as int64_t/int32_t