[IMPALA-12002] COMMIT_COMPACTION_EVENT are not correctly skipped if the db not found

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: Impala 4.3.0
• Epic Link: event-processor-completeness
• Epic Color: ghx-label-12
• Assignee: Sai Hemanth Gantasala
• Reporter: Quanlong Huang
• Created: 15/Mar/23 02:29
• Updated: 12/May/23 02:14
• Resolved: 23/Mar/23 07:31

Description

I saw event-processor is stopped due to an error in a CORE build: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/18906/

I0313 02:30:30.313375 30039 MetastoreEventsProcessor.java:805] Received 8 events. Start event id : 26504
I0313 02:30:30.313757 30039 MetastoreEventsProcessor.java:1026] Time elapsed in processing event batch: 283.896us
E0313 02:30:30.313935 30039 MetastoreEventsProcessor.java:865] Unexpected exception received while processing event
Java exception follows:
org.apache.impala.catalog.events.MetastoreNotificationException: EventId: 26506 EventType: COMMIT_COMPACTION_EVENT Unable to parse commit compaction message
        at org.apache.impala.catalog.events.MetastoreEvents$CommitCompactionEvent.<init>(MetastoreEvents.java:2684)
        at org.apache.impala.catalog.events.MetastoreEvents$MetastoreEventFactory.get(MetastoreEvents.java:223)
        at org.apache.impala.catalog.events.MetastoreEvents$MetastoreEventFactory.getFilteredEvents(MetastoreEvents.java:255)
        at org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:999)
        at org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:851)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.impala.catalog.DatabaseNotFoundException: Database 'test_acid_compaction_5c682e9f' not found
        at org.apache.impala.catalog.Catalog.getTable(Catalog.java:196)
        at org.apache.impala.catalog.events.MetastoreEvents$CommitCompactionEvent.<init>(MetastoreEvents.java:2679)
        ... 11 more
E0313 02:30:30.314000 30039 MetastoreEventsProcessor.java:1040] Notification event is null
W0313 02:30:31.314235 30039 MetastoreEventsProcessor.java:844] Event processing is skipped since status is ERROR. Last synced event id is 26504

"test_acid_compaction_5c682e9f" is a temp db created by tests. It's already dropped when catalogd received the commit compaction event. The exception is thrown from the constructor of CommitCompactionEvent. We should skip the event instead of let it fails the event-processor.

Issue Links

• is related to: IMPALA-11968 test_event_based_replication fails with timeout
• is related to: IMPALA-11352 CI: Transient failure in test_event_based_replication
• relates to: IMPALA-11626 Handle COMMIT_COMPACTION_EVENT from HMS