[GOBBLIN-861] Skip getPartition() call to Hive Metastore when a partition already exists

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.15.0
• Component/s: hive-registration
• Assignee: Abhishek Tiwari
• Reporter: Zihan Li
• Created: 16/Aug/19 01:12
• Updated: 03/Aug/20 19:09
• Resolved: 24/Jul/20 21:27

Description

Currently, we compute a diff between the current partition and an already registered partition when a partition has already been registered in Hive. This is done by calling getPartition() on the Hive metastore client, which can be expensive. Since no time-varying attributes are stored in a Hive partition, diff computation (and getPartition() call) can be skipped when a partition already exists.

Issue Links

• links to: GitHub Pull Request #2716