[ATLAS-2889] S3 object tag import hook

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.1.0
• Component/s: atlas-core, atlas-intg
• Assignee: Unassigned
• Reporter: t oo
• Created: 25/Sep/18 00:00
• Updated: 26/Sep/18 05:47

Description

https://issues.apache.org/jira/browse/ATLAS-2708 introduced support for s3 type.

 

From the comments in the jira, there are some external steps needed to import object tags. This Jira is about creating the E2E functionality within Atlas. ie functionality in UI or an API that when supplied with argument of an s3 bucket/slice then Atlas automatically imports all the object names/tags recursively from under that s3 bucket/slice.

 

"It doesn't do it automatically through a listener like the hive hook.  We do it via lambda functions, triggered, say, on the creation of S3 object or pseudodirectory or bucket.  We package up the info into AtlasEntities and then publish to the ATLAS_HOOK kafka topic.

You have to create your own Lambda code that creates AtlasEntities on the fly as trigerred by Lambda Function(on changes made to s3 object) and then push to Kafka Queue. This particular functionality is not part of Atlas tool as of now."

cc: barbara ayushmnnit

Issue Links

• is related to: ATLAS-2708 AWS S3 data lake typedefs for Atlas