[ATLAS-3148] Implement Hive Metastore hook for Atlas

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 2.0.0, trunk
• Component/s: atlas-core
• Labels: new-feature
• Assignee: Sarath Subramanian
• Reporter: Sarath Subramanian
• Created: 17/Apr/19 23:17
• Updated: 09/Dec/23 21:23
• Resolved: 17/Apr/19 23:50

Description

Atlas has hive hook (http://atlas.apache.org/Hook-Hive.html) which registers with HiveServer2 process to listen for create/update/delete operations and updates the metadata in Atlas.

If hive metastore is accessed using other clients like Impala shell, Hue or other JDBC/ODBC client apps there is no way to capture the events in Atlas.

This Jira will create a new atlas hook for Hive Metastore - stores the metadata for Hive tables and partitions in a relational database, and provides clients (including Hive) access to this information using the metastore service API. 

This hook is registered as post-listener (hive.metastore.event.listeners) and DDL operations are captured and send to atlas kafka topic for processing by Atlas server.

The following DDL operations are captured:

• CreateDatabaseEvent
• DropDatabaseEvent
• AlterDatabaseEvent
• CreateTableEvent
• DropTableEvent
• AlterTableEvent

 

Issue Links

• is duplicated by: ATLAS-2779 Create Hive Metastore Event listener to push table changes to Atlas.