[HUDI-5319] NPE in Bloom Filter Index

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Duplicate
• Affects Version/s: 0.12.1
• Fix Version/s: 0.14.0
• Story Points: 0
• Epic Link: Harden read/write core
• Assignee: Ethan Guo
• Reporter: Alexey Kudinkin

Reviewers

sivabalan narayanan

• Created: 02/Dec/22 17:29
• Updated: 23/May/23 05:27
• Resolved: 19/Jan/23 19:04

Description

/12/02 11:05:49 WARN TaskSetManager: Lost task 3.0 in stage 1098.0 (TID 1300185) (ip-172-31-23-246.us-east-2.compute.internal executor 10): java.lang.RuntimeException: org.apache.hudi.exception.HoodieIndexException: Error checking bloom filter index.
        at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:121)
        at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:46)
        at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
        at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:513)
        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:491)
        at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:183)
        at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
        at org.apache.spark.scheduler.Task.run(Task.scala:138)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1516)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hudi.exception.HoodieIndexException: Error checking bloom filter index.
        at org.apache.hudi.index.bloom.HoodieBloomIndexCheckFunction$LazyKeyCheckIterator.computeNext(HoodieBloomIndexCheckFunction.java:110)
        at org.apache.hudi.index.bloom.HoodieBloomIndexCheckFunction$LazyKeyCheckIterator.computeNext(HoodieBloomIndexCheckFunction.java:60)
        at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:119)
        ... 16 more
Caused by: java.lang.NullPointerException
        at org.apache.hudi.io.HoodieKeyLookupHandle.addKey(HoodieKeyLookupHandle.java:87)
        at org.apache.hudi.index.bloom.HoodieBloomIndexCheckFunction$LazyKeyCheckIterator.computeNext(HoodieBloomIndexCheckFunction.java:92)
        ... 18 more 

Issue Links

• is broken by: HUDI-5323 Decouple virtual key with writing bloom filters to parquet files
• is duplicated by: HUDI-5323 Decouple virtual key with writing bloom filters to parquet files