Uploaded image for project: 'Falcon'
  1. Falcon
  2. FALCON-285

Support Lineage information capture

Add voteWatch issue
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.5
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      We would want to capture enough information from entities and the associated executions to drive lineage tracing and visualization.
      The plan is to capture lineage specific information - which is all the inputs, process and associated workflows, outputs for each execution in Falcon post processing step. Post message consumption, this information is acted upon and persisted in falcon server for each output generated. This should work for replication as well. I need to think about eviction which was not invoking post processing but will soon.

        Attachments

          Issue Links

          1.
          Capture information in process entity about the user workflow Sub-task Resolved Venkatesh Seetharam Actions
          2.
          Record lineage information in post processing Sub-task Resolved Venkatesh Seetharam Actions
          3.
          Persist lineage information into a persistent store Sub-task Resolved Venkatesh Seetharam Actions
          4.
          Provide REST APIs for discovering lineage metadata over the store Sub-task Resolved Venkatesh Seetharam Actions
          5.
          Visualize lineage information on the dashboard Sub-task Closed Haohui Mai Actions
          6.
          Document lineage feature Sub-task Resolved Sowmya Ramesh Actions
          7.
          Process lineage information for Replication policies Sub-task Resolved Sowmya Ramesh Actions
          8.
          Add indexing to the graph property keys Sub-task Resolved Venkatesh Seetharam Actions
          9.
          Clean up historical data periodically Sub-task Open Unassigned Actions
          10.
          Add existing entities from store at startup if graphed is empty for backwards compatibility Sub-task Resolved Ajay Yadav Actions
          11.
          Bug when MetadataMappingService is not configured as one of the application services Sub-task Resolved Venkatesh Seetharam Actions
          12.
          REST API does not conform to Rexster Sub-task Resolved Venkatesh Seetharam Actions
          13.
          Instance id's captured are of different formats in process and feed Sub-task Resolved Venkatesh Seetharam Actions
          14.
          Lineage recording fails with NPE for processes with >1 inputs Sub-task Resolved Venkatesh Seetharam Actions
          15.
          Add a REST API to get properties for a given vertex Sub-task Resolved Venkatesh Seetharam Actions
          16.
          Remove Graph dump option in CLI Sub-task Resolved Venkatesh Seetharam Actions
          17.
          Show vertex information in the web UI Sub-task Closed Haohui Mai Actions
          18.
          Display lineage link only for jobs that are succeeded in the web UI Sub-task Closed Haohui Mai Actions
          19.
          Lineage breaks if feed.xml doesn't have the date pattern in feed path location Sub-task Resolved Sowmya Ramesh Actions
          20.
          Preserve data type for properties in a vertex Sub-task Resolved Ajay Yadav Actions
          21.
          Process lineage information for Retention policies Sub-task Resolved Sowmya Ramesh Actions
          22.
          Enable metrics for Titan Sub-task Closed Ajay Yadav Actions
          23.
          Upgrade Blueprints to latest and Titan to latest (0.5) Sub-task Open Unassigned Actions

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                svenkat Venkatesh Seetharam

                Dates

                • Created:
                  Updated:

                  Issue deployment