[IMPALA-12210] In external frontend mode, catalogd.INFO shows get_table_req() forwarded to HMS and an exception

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.3.0
• Component/s: Catalog
• Epic Color: ghx-label-2
• Assignee: Sai Hemanth Gantasala
• Reporter: Sai Hemanth Gantasala
• Created: 13/Jun/23 17:16
• Updated: 26/Jun/23 17:42
• Resolved: 26/Jun/23 17:42

Description

For queries submitted through external frontend, select  on external tables shows an exception in the catalogd.INFO log and catalogd forwards the get_table_req() requests to HMS (this is on the second execution of the query):

0: jdbc:hive2://localhost:11050/default> select * from ext_tbl_t1;

I0503 15:12:54.733690 15625 CatalogMetastoreServer.java:215] Invoking HMS API: open_txns
I0503 15:12:54.746543 15625 CatalogMetastoreServer.java:215] Invoking HMS API: get_open_txns_req
I0503 15:12:54.753791 15625 CatalogMetastoreServer.java:215] Invoking HMS API: get_valid_write_ids
I0503 15:12:54.757498 15625 CatalogMetastoreServer.java:215] Invoking HMS API: get_table_req
I0503 15:12:54.757787 15625 MetastoreServiceHandler.java:1506] Received exception while executing get_table_req
Java exception follows:
java.lang.IllegalStateException: Compaction id check cannot be done for non-transactional table foo.ext_tbl_t1  at com.google.common.base.Preconditions.checkState(Preconditions.java:508)
	at org.apache.impala.catalog.CatalogServiceCatalog.getOrLoadTable(CatalogServiceCatalog.java:2247)
	at org.apache.impala.catalog.CatalogServiceCatalog.doGetPartialCatalogObject(CatalogServiceCatalog.java:3537)
	at org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:3463)
	at org.apache.impala.catalog.CatalogHmsAPIHelper.getPartialCatalogObjResponse(CatalogHmsAPIHelper.java:216)
	at org.apache.impala.catalog.CatalogHmsAPIHelper.getTableReq(CatalogHmsAPIHelper.java:152)
	at org.apache.impala.catalog.metastore.CatalogMetastoreServiceHandler.get_table_req(CatalogMetastoreServiceHandler.java:114)
	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.impala.catalog.metastore.CatalogMetastoreServer$TimingInvocationHandler.invoke(CatalogMetastoreServer.java:216)
	at com.sun.proxy.$Proxy13.get_table_req(Unknown Source)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:20036)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:20015)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
I0503 15:12:54.757825 15625 MetastoreServiceHandler.java:897] Forwarding the request get_table_req for table db1.tt3 to the backing HiveMetastore service

For external tables, we don't need to do the compaction id check. Also, after the first execution, the expectation is that the table metadata is cached in the catalogd so the get_table_req() should ideally be satisfied from the cached entries instead of being forwarded to HMS.