Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-4993

ReportLineageToAtlas complete path strategy does not report some lineages with secured NiFi

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.0
    • Fix Version/s: 1.6.0
    • Component/s: Extensions
    • Labels:
      None

      Description

      ReportLineageToAtlas 'complete path' strategy uses NiFi provenance lineage query with an anonymous user. If NiFi is secured and the user who made the lineage query request does not have required privilege, NiFi returns provenance event type as UNKNOWN, and also does not traverse lineage fully.

      Specifically, the authorization is implemented here:
      https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L2641

      PersistentProvenanceRepository$ComputeLineageRunnable.run
      final StandardLineageResult result = submission.getResult();
      result.update(replaceUnauthorizedWithPlaceholders(matchingRecords, user), matchingRecords.size());
      

      This affects to ReportLineageToAtlas 'complete path' strategy as it will not be able to traverse parent provenance events to analyze full lineage path for a FlowFile. As a result, the reporting task can not report lineage with some structures of flow.

       For example, with the following NiFi flow, the FlowFile that was RECEIVEd by GetFile went through Kafka route (the right branch). Also, the FlowFile was CLONEd to go Hive and HDFS routes.

      Then the original FlowFile that went through Kafka route would have NiFi lineage like this. This lineage can be retrieved by single lineage query and works even with an anonymous user. These routes can be reported to Atlas:

      However, the CLONEd routes would have following lineage. This graph was queried from NiFi UI by a NiFi user who has sufficient privilege. But with an anonymous user, the link from SEND (23) to the FlowFile then CLONE (18) is not returned. Because event types are masked as UNKNOWN and NiFi framework does not traverse the linkage. Thus, these cloned routes are not reported to Atlas.

      ReportLineageToAtlas needs to have a property so that user can specify a NiFi user id to impersonate, so that required policies can be administrated. 1st PR 2567
      Instead of letting user to specify a NiFi user id, the updated 2nd PR (2577) fixes lineage computation with unauthorized user.
      The 3rd PR (2589) attempts fixing this issue by modifying ProvenanceRepository implementations to accept null user so that lineage query can be called by NiFi internal components.

      This issue was originally reported by Mahesh Nayak.

      A simplified NiFi flow template to test the proposed fix is attached.

        Attachments

        1. flow-screenshot.png
          856 kB
          Koji Kawamura
        2. hdfs-route.png
          29 kB
          Koji Kawamura
        3. kafka-route.png
          35 kB
          Koji Kawamura
        4. NIFI-4993.xml
          69 kB
          Koji Kawamura
        5. unauthorized-query.png
          19 kB
          Koji Kawamura
        6. unauthorized-query-with-fix.png
          11 kB
          Koji Kawamura

          Activity

            People

            • Assignee:
              ijokarumawak Koji Kawamura
              Reporter:
              ijokarumawak Koji Kawamura
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: