Uploaded image for project: 'Atlas'
  1. Atlas
  2. ATLAS-3661

Create 'spark_column_lineage' type and relationship definition to add support of column level lineage

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.1.0, 3.0.0
    • None
    • None

    Description

      Create 'spark_column_lineage' type and corresponding 'spark_process_column_lineage' relationship definition to add support of column-level lineage.

      Column level lineage refers to lineage created between the input and output columns.
      For example:

      hive > create table employee_ctas as select id from employee;

      For the above query, lineage is created from 'employee' to 'employee_ctas',
      and also from 'employee.id' to 'employee_ctas.id'.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            vladglinskiy Vladislav Glinskiy
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h 40m
                1h 40m

                Slack

                  Issue deployment