[IMPALA-12053] Expose event-processor error message in catalogd WebUI

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.3.0
• Component/s: Catalog
• Labels: observability
• Epic Link: event-processor-supportability
• Epic Color: ghx-label-8
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 11/Apr/23 01:14
• Updated: 07/Mar/24 19:45
• Resolved: 26/May/23 02:47

Description

Currently, we can only check logs to know why event-processor comes into ERROR state. E.g. here is the error message for IMPALA-12002:

E0315 17:54:14.959136  2659 MetastoreEventsProcessor.java:865] Unexpected exception received while processing event
Java exception follows:
org.apache.impala.catalog.events.MetastoreNotificationException: EventId: 26044 EventType: COMMIT_COMPACTION_EVENT Unable to parse commit compaction message
        at org.apache.impala.catalog.events.MetastoreEvents$CommitCompactionEvent.<init>(MetastoreEvents.java:2684)
        at org.apache.impala.catalog.events.MetastoreEvents$MetastoreEventFactory.get(MetastoreEvents.java:223)
        at org.apache.impala.catalog.events.MetastoreEvents$MetastoreEventFactory.getFilteredEvents(MetastoreEvents.java:255)
        at org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:999)
        at org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:851)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.impala.catalog.DatabaseNotFoundException: Database 'test_acid_compaction_5c682e9f' not found
        at org.apache.impala.catalog.Catalog.getTable(Catalog.java:196)
        at org.apache.impala.catalog.events.MetastoreEvents$CommitCompactionEvent.<init>(MetastoreEvents.java:2679)
        ... 11 more
E0315 17:54:14.959192  2659 MetastoreEventsProcessor.java:1040] Notification event is null
W0315 17:54:15.959450  2659 MetastoreEventsProcessor.java:844] Event processing is skipped since status is ERROR. Last synced event id is 26042 

It'd be helpful to expose the stacktrace in catalogd's WebUI. Tests can also retrieve it to show more detailed failures instead of just complaining timeout.

Issue Links

• causes: IMPALA-12170 TestWebPage.test_webserver_hide_logs_link fails due to new /events link added in catalogd
• relates to: IMPALA-7975 Improve supportability of the automatic invalidate feature