[NIFI-6945] Use minExtInfo=true to reduce the amount of data queried from Atlas

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.11.0
• Component/s: Extensions
• Assignee: Peter Turcsanyi
• Reporter: Peter Turcsanyi
• Created: 12/Dec/19 14:19
• Updated: 10/Jan/20 14:05
• Resolved: 10/Jan/20 14:05

Description

When ReportLineageToAtlas queries the entities from Atlas, the AtlasEntityWithExtInfo response object also contains all the referred entities with all their data by default which leads to huge data transfers between Atlas and NiFi. On the other hand, this extended data of the related entities are not used by the reporting task, so they could be excluded from the query (using minExtInfo=true parameter) in order to reduce the response size.

Issue Links

• depends upon: NIFI-6939 Upgrade Atlas client to 2.0.0
• links to: GitHub Pull Request #3970