I am trying to trigger upserts on a table that has an array field with records of just one field.
Here is the code to reproduce:
Running insert following by upsert will fail:
If I create the books record with all the fields (at least 2), it works as expected.
The relevant part of the exception is this:
Another way to test is by changing the generated data in the tips to just the amount, by dropping the currency on the tips_history field, tests will start failing:
I have narrowed this down to this block in the parquet-avro integration: https://github.com/apache/parquet-mr/blob/master/parquet-avro/src/main/java/org/apache/parquet/avro/AvroRecordConverter.java#L846-L875
Which always returns false after trying to decide whether reader and writer schemas are compatible. Going through that code path makes me think it's related to the fields being optional, as the inferred schema seems to be (null, string) with default null instead of (string, null) with no default.
At this point I'm lost, tried to figure something out based on this https://github.com/apache/hudi/pull/1406/files but I'm not sure where to start.