[IMPALA-6671] Metadata operations that modify a table blocks topic updates for other unrelated operations

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.10.0, Impala 2.11.0, Impala 2.12.0
• Fix Version/s: Impala 4.0.0
• Component/s: Catalog
• Labels: catalog-server performance
• Target Version: Impala 4.0.0
• Epic Color: ghx-label-5
• Assignee: Vihang Karajgaonkar
• Reporter: Mostafa Mokhtar
• Created: 15/Mar/18 01:08
• Updated: 07/Feb/23 11:14
• Resolved: 23/Dec/20 23:52

Description

Metadata operations that mutate the state of a table like "compute stats foo" or "alter recover partitions" block topic updates for read only operations against unrelated tables as "describe bar".

Thread for blocked operation

"Thread-7" prio=10 tid=0x0000000011613000 nid=0x21b3b waiting on condition [0x00007f5f2ef52000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f6f57ff0240> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
        at org.apache.impala.catalog.CatalogServiceCatalog.addTableToCatalogDeltaHelper(CatalogServiceCatalog.java:639)
        at org.apache.impala.catalog.CatalogServiceCatalog.addTableToCatalogDelta(CatalogServiceCatalog.java:611)
        at org.apache.impala.catalog.CatalogServiceCatalog.addDatabaseToCatalogDelta(CatalogServiceCatalog.java:567)
        at org.apache.impala.catalog.CatalogServiceCatalog.getCatalogDelta(CatalogServiceCatalog.java:449)
        at org.apache.impala.service.JniCatalog.getCatalogDelta(JniCatalog.java:126)

Thread for blocking operation

"Thread-130" prio=10 tid=0x00000000113d5800 nid=0x2499d runnable [0x00007f5ef80d0000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:152)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        - locked <0x00007f5fffcd9f18> (a java.io.BufferedInputStream)
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
        at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:346)
        at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:423)
        at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:405)
        at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
        at org.apache.hadoop.hive.thrift.TFilterTransport.readAll(TFilterTransport.java:62)
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_add_partitions_req(ThriftHiveMetastore.java:1639)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.add_partitions_req(ThriftHiveMetastore.java:1626)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.add_partitions(HiveMetaStoreClient.java:609)
        at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:101)
        at com.sun.proxy.$Proxy10.add_partitions(Unknown Source)
        at org.apache.impala.service.CatalogOpExecutor.alterTableRecoverPartitions(CatalogOpExecutor.java:2651)
        at org.apache.impala.service.CatalogOpExecutor.alterTable(CatalogOpExecutor.java:525)
        at org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:262)
        at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:146)

Issue Links

• is blocked by: IMPALA-6843 Responses to prioritizedLoad() requests should be returned directly and not via the statestore
• is broken by: IMPALA-5058 Improve concurrency of DDL/DML operations during catalog updates
• is duplicated by: IMPALA-8301 Eliminate need for SYNC_DDL in local catalog mode
• is related to: IMPALA-8301 Eliminate need for SYNC_DDL in local catalog mode

Sub-Tasks

• Add a query option to simulate catalogd HDFS listing delays: Resolved