[HUDI-722] IndexOutOfBoundsException in MessageColumnIORecordConsumer.addBinary when writing parquet

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.9.0
• Component/s: writer-core
• Assignee: sivabalan narayanan
• Reporter: Alexander Filipchik
• Created: 19/Mar/20 02:24
• Updated: 25/Mar/22 15:21

Description

Some writes fail with java.lang.IndexOutOfBoundsException : Invalid array range: X to X inside MessageColumnIORecordConsumer.addBinary call.

Specifically: getColumnWriter().write(value, r[currentLevel], currentColumnIO.getDefinitionLevel());

fails as size of r is the same as current level. What can be causing it?

 

It gets executed via: ParquetWriter.write(IndexedRecord) Library version: 1.10.1 Avro is a very complex object (~2.5k columns, highly nested, arrays of unions present).

But what is surprising is that it fails to write top level field: PrimitiveColumnIO _hoodie_commit_time r:0 d:1 [_hoodie_commit_time] which is the first top level field in Avro: {"_hoodie_commit_time": "20200317215711", "_hoodie_commit_seqno": "20200317215711_0_650",

Issue Links

• is depended upon by: HUDI-901 Bug Bash 0.6.0 Tracking Ticket