[IMPALA-13126] ReloadEvent.isOlderEvent() should hold the table read lock

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Component/s: Catalog
• Labels: catalog-2024
• Epic Link: event-processor-completeness
• Epic Color: ghx-label-14
• Assignee: Sai Hemanth Gantasala
• Reporter: Quanlong Huang
• Created: 03/Jun/24 06:50
• Updated: 10/Jun/24 17:00

Description

Saw an exception like this:

E0601 09:11:25.275251   246 MetastoreEventsProcessor.java:990] Unexpected exception received while processing event
Java exception follows:
java.util.ConcurrentModificationException
        at java.util.HashMap$HashIterator.nextNode(HashMap.java:1469)
        at java.util.HashMap$ValueIterator.next(HashMap.java:1498)
        at org.apache.impala.catalog.FeFsTable$Utils.getPartitionFromThriftPartitionSpec(FeFsTable.java:616)
        at org.apache.impala.catalog.HdfsTable.getPartitionFromThriftPartitionSpec(HdfsTable.java:597)
        at org.apache.impala.catalog.Catalog.getHdfsPartition(Catalog.java:511)
        at org.apache.impala.catalog.Catalog.getHdfsPartition(Catalog.java:489)
        at org.apache.impala.catalog.CatalogServiceCatalog.isPartitionLoadedAfterEvent(CatalogServiceCatalog.java:4024)
        at org.apache.impala.catalog.events.MetastoreEvents$ReloadEvent.isOlderEvent(MetastoreEvents.java:2754)
        at org.apache.impala.catalog.events.MetastoreEvents$ReloadEvent.processTableEvent(MetastoreEvents.java:2729)
        at org.apache.impala.catalog.events.MetastoreEvents$MetastoreTableEvent.process(MetastoreEvents.java:1107)
        at org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:531)
        at org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:1164)
        at org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:972)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750) 

For a partition-level RELOAD event, ReloadEvent.isOlderEvent() needs to check whether the corresponding partition is reloaded after the event. This should be done after holding the table read lock. Otherwise, EventProcessor could hit the error above when there are concurrent DDLs/DMLs modifying the partition list.

CC VenuReddy

Issue Links

• is caused by: IMPALA-11822 Optimize the Refresh/Invalidate event processing by skipping unnecessary events