[HIVE-7195] Improve Metastore performance

• Type: Improvement
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Component/s: Metastore
• Assignee: Unassigned
• Reporter: Brock Noland
• Created: 08/Jun/14 21:15
• Updated: 01/Oct/19 22:07

Description

Even with direct SQL, which significantly improves MS performance, some operations take a considerable amount of time, when there are many partitions on table. Specifically I believe the issue:

• When a client gets all partitions we do not send them an iterator, we create a collection of all data and then pass the object over the network in total

• Operations which require looking up data on the NN can still be slow since there is no cache of information and it's done in a serial fashion

• Perhaps a tangent, but our client timeout is quite dumb. The client will timeout and the server has no idea the client is gone. We should use deadlines, i.e. pass the timeout to the server so it can calculate that the client has expired.

Issue Links

• contains: HIVE-6980 Drop table by using direct sql
• relates to: HIVE-6179 OOM occurs when query spans to a large number of partitions
• relates to: HIVE-7140 Bump default hive.metastore.client.socket.timeout to 5 minutes

Sub-Tasks

• Support Postgres via DirectSQL: Open
• MetaStore server should support timeout for long running requests: Closed
• Add timeout check inside the HMS server: Closed
• Add timeout mechanism in RawStoreProxy: Closed