[ATLAS-3006] Option to ignore/prune metadata for temporary/staging Hive tables

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.8.4, 1.2.0, 2.0.0
• Component/s: atlas-core
• Labels: new-feature
• Assignee: Madhan Neethiraj
• Reporter: Madhan Neethiraj
• Created: 24/Dec/18 20:15
• Updated: 26/Sep/19 00:38
• Resolved: 04/Jan/19 22:09

Description

It is not uncommon for a Hive deployment to use a large number of staging/temporary tables, which are created periodically to load data into target tables and deleted after completion of data load. A large number of entities are created in Atlas for these staging/temporary tables (tables/columns/column-lineage).

For staging tables, it is probably not useful to track details like columns and column-lineage in Atlas. Not tracking these details in Atlas can significantly reduce the time it takes to process notifications, and can help in improving the performance overall. Only minimum details of these staging tables can be stored in Atlas, to capture data lineage from source to target table via all intermediate staging tables.

Also, it will be helpful to good to ignore tables that are created & deleted during data loading i.e. temporary tables.

Configurations should be provided to specify which of the tables are staging/temporary. In addition to supporting this in Hive hook (to avoid generation of large messages for staging/temporary tables), Atlas server should also be updated, to control this further at server side while processing notifications.

Issue Links

• contains: ATLAS-3085 Provide an option in atlas to disable tracking of specified hive table entities and its lineages
• links to: Patch for master branch in review board