[HIVE-27260] OOM while retrieving delegation tokens using get_all_token_identifiers call

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Hive, Standalone Metastore
• Assignee: Unassigned
• Reporter: Taraka Rama Rao Lethavadla
• Created: 13/Apr/23 15:14
• Updated: 13/Apr/23 15:21

Description

get_all_token_identifiers is retrieving all the entries in table DELEGATION_TOKENS at once.

In systems where the total number of rows is very high causes OOM in HMS while retrieving all the rows at once.

Can we have batching kind of mechanism while retrieving rows from that table?

How about we add timestamp column to that table so that old entries can be cleaned up manually incase going via cleaner did not help?

Even expiry token removal thread is also using the same api get_all_token_identifiers and it becomes difficult for cleaner once the number of rows reaches higher number

 

There is a feature https://issues.apache.org/jira/browse/HIVE-17609 but that is also using get_all_token_identifiers, so it will also lead to the same issue