Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Invalid
-
0.9.0
Description
The hoodie table h0's schema is :
(id long, price double)
when I write the dataframe to h0 with the follow schema:
(id long, price int)
An Exception is threw as follow:
at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:136)at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:136) at org.apache.hudi.common.util.ParquetReaderIterator.hasNext(ParquetReaderIterator.java:49) at org.apache.hudi.common.util.queue.IteratorBasedQueueProducer.produce(IteratorBasedQueueProducer.java:45) at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$0(BoundedInMemoryExecutor.java:102) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ... 4 moreCaused by: java.lang.UnsupportedOperationException: org.apache.parquet.avro.AvroConverters$FieldIntegerConverter at org.apache.parquet.io.api.PrimitiveConverter.addDouble(PrimitiveConverter.java:84) at org.apache.parquet.column.impl.ColumnReaderImpl$2$2.writeValue(ColumnReaderImpl.java:228) at org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:367) at org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:406) at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:226) ... 11 more
I have enable the AVRO_SCHEMA_VALIDATE, it can pass the schema validate in HoodieTable#validateUpsertSchema, so it is right to write the "int" data to the "double" field in hoodie.
Attachments
Issue Links
- is related to
-
HUDI-741 Fix Hoodie's schema evolution checks
- Resolved
- links to