Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-3398

Schema validation fails for metadata table base file

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      Stacktrace:

      java.lang.IllegalArgumentException: Unknown file format :file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/files-0000_0-93-815_20220208164926830001.hfile	at org.apache.hudi.common.table.TableSchemaResolver.getTableParquetSchemaFromDataFile(TableSchemaResolver.java:103)	at org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchemaFromDataFile(TableSchemaResolver.java:119)	at org.apache.hudi.common.table.TableSchemaResolver.hasOperationField(TableSchemaResolver.java:480)	at org.apache.hudi.common.table.TableSchemaResolver.<init>(TableSchemaResolver.java:65)	at org.apache.hudi.table.HoodieTable.validateSchema(HoodieTable.java:682)	at org.apache.hudi.table.HoodieTable.validateUpsertSchema(HoodieTable.java:698)	at org.apache.hudi.client.SparkRDDWriteClient.upsertPreppedRecords(SparkRDDWriteClient.java:173)	at org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.commit(SparkHoodieBackedTableMetadataWriter.java:154)	at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.processAndCommit(HoodieBackedTableMetadataWriter.java:663)	at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java:675)	at org.apache.hudi.client.BaseHoodieWriteClient.lambda$writeTableMetadata$0(BaseHoodieWriteClient.java:273)	at org.apache.hudi.common.util.Option.ifPresent(Option.java:96)	at org.apache.hudi.client.BaseHoodieWriteClient.writeTableMetadata(BaseHoodieWriteClient.java:273)	at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:229)	at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:199)	at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:127)	at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:609)	at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:329)	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:652)	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)	at java.lang.Thread.run(Thread.java:748) 

      full logs: https://gist.github.com/yihua/e00a1caddacbdc570b5b757049750f39

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            zhangyue19921010 Yue Zhang
            guoyihua Ethan Guo
            Ethan Guo
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Agile

                Completed Sprint:
                Hudi-Sprint-Feb-7 ended 15/Feb/22
                View on Board

                Slack

                  Issue deployment