[HUDI-3453] Metadata table throws NPE when scheduling compaction plan

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.12.2
• Component/s: core
• Labels: pull-request-available
• Story Points: 0.5
• Epic Link: Metadata Table for File Listing & Query Planning
• Assignee: Yue Zhang
• Reporter: Danny Chen

Reviewers

Ethan Guo

• Created: 18/Feb/22 03:57
• Updated: 19/Dec/22 13:35
• Resolved: 19/Dec/22 13:35

Description

Caused by: org.apache.hudi.exception.HoodieException: Error occurs when executing flatMap
  at org.apache.hudi.common.function.FunctionWrapper.lambda$throwingFlatMapWrapper$1(FunctionWrapper.java:50)
  at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:269)
  at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
  at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
  at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
  at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747)
  at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721)
  at java.util.stream.AbstractTask.compute(AbstractTask.java:327)
  at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
  at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
  at java.util.concurrent.ForkJoinPool.helpComplete(ForkJoinPool.java:1870)
  at java.util.concurrent.ForkJoinPool.externalHelpComplete(ForkJoinPool.java:2467)
  at java.util.concurrent.ForkJoinTask.externalAwaitDone(ForkJoinTask.java:324)
  at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:405)
  at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
  at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:714)
  at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
  at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
  at org.apache.hudi.client.common.HoodieFlinkEngineContext.flatMap(HoodieFlinkEngineContext.java:136)
  at org.apache.hudi.table.action.compact.HoodieCompactor.generateCompactionPlan(HoodieCompactor.java:263)
  at org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor.scheduleCompaction(ScheduleCompactionActionExecutor.java:122)
  at org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor.execute(ScheduleCompactionActionExecutor.java:92)
  at org.apache.hudi.table.HoodieFlinkMergeOnReadTable.scheduleCompaction(HoodieFlinkMergeOnReadTable.java:109)
  at org.apache.hudi.client.AbstractHoodieWriteClient.scheduleTableServiceInternal(AbstractHoodieWriteClient.java:1100)
  at org.apache.hudi.client.AbstractHoodieWriteClient.scheduleTableService(AbstractHoodieWriteClient.java:1083)
  at org.apache.hudi.client.AbstractHoodieWriteClient.scheduleCompactionAtInstant(AbstractHoodieWriteClient.java:850)
  at org.apache.hudi.client.AbstractHoodieWriteClient.scheduleCompaction(AbstractHoodieWriteClient.java:841)
  at org.apache.hudi.util.CompactionUtil.scheduleCompaction(CompactionUtil.java:64)
  at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$notifyCheckpointComplete$2(StreamWriteOperatorCoordinator.java:229)
  at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$execute$0(NonThrownExecutor.java:93)
  ... 3 more
Caused by: org.apache.hudi.exception.HoodieMetadataException: Failed to retrieve files in partition oss://datalake-huifu/hudi/poc/ods/pnrweb_prod/trans_log/20220216 from metadata
  at org.apache.hudi.metadata.BaseTableMetadata.getAllFilesInPartition(BaseTableMetadata.java:124)
  at org.apache.hudi.metadata.HoodieMetadataFileSystemView.listPartition(HoodieMetadataFileSystemView.java:65)
  at org.apache.hudi.common.table.view.AbstractTableFileSystemView.lambda$ensurePartitionLoadedCorrectly$9(AbstractTableFileSystemView.java:304)
  at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
  at org.apache.hudi.common.table.view.AbstractTableFileSystemView.ensurePartitionLoadedCorrectly(AbstractTableFileSystemView.java:295)
  at org.apache.hudi.common.table.view.AbstractTableFileSystemView.getLatestFileSlices(AbstractTableFileSystemView.java:578)
  at org.apache.hudi.common.table.view.PriorityBasedFileSystemView.execute(PriorityBasedFileSystemView.java:80)
  at org.apache.hudi.common.table.view.PriorityBasedFileSystemView.getLatestFileSlices(PriorityBasedFileSystemView.java:170)
  at org.apache.hudi.table.action.compact.HoodieCompactor.lambda$generateCompactionPlan$30498406$1(HoodieCompactor.java:264)
  at org.apache.hudi.common.function.FunctionWrapper.lambda$throwingFlatMapWrapper$1(FunctionWrapper.java:48)
  ... 32 more
Caused by: java.lang.NullPointerException
  at org.apache.hudi.io.storage.HoodieHFileReader.getRecordByKey(HoodieHFileReader.java:249)
  at org.apache.hudi.io.storage.HoodieFileReader.getRecordByKey(HoodieFileReader.java:49)
  at org.apache.hudi.metadata.HoodieBackedTableMetadata.readFromBaseAndMergeWithLogRecords(HoodieBackedTableMetadata.java:195)
  at org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordsByKeys(HoodieBackedTableMetadata.java:141)
  at org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordByKey(HoodieBackedTableMetadata.java:124)
  at org.apache.hudi.metadata.BaseTableMetadata.fetchAllFilesInPartition(BaseTableMetadata.java:184)
  at org.apache.hudi.metadata.BaseTableMetadata.getAllFilesInPartition(BaseTableMetadata.java:122)
  ... 41 more

Issue Links

• links to: GitHub Pull Request #5091