[HIVE-20633] Incorrect column lineage: each output column has input from *all columns* of the input table

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 1.2.2
• Component/s: HiveServer2
• Assignee: Unassigned
• Reporter: Madhan Neethiraj
• Created: 25/Sep/18 21:35
• Updated: 05/Nov/18 21:00

Description

Column lineage details made available to post hook is incorrect for certain queries - like the following INSERT:

CREATE TABLE source_tbl(col_001 INT, col_002 INT, col_003 INT);

CREATE TABLE target_tbl(col_001 INT, col_002 INT, col_003 INT);

INSERT INTO target_tbl SELECT v1.col_001, v1.col_002, v1.col_003 FROM (SELECT col_001, col_002, col_003, ROW_NUMBER() OVER() AS r_num FROM source_tbl) v1;

Below are the details of the lineage given to post hooks (like Atlas hook) via HookContext.getLinfo(). It contains 3 entries, one for each target table column. Note the dependency for each column has all columns of the source tables.

DependencyKey=default.target_tbl:FieldSchema(name:col_001, type:int, comment:null)
Dependency=[SCRIPT]
           [default.source_tbl(src):FieldSchema(name:col_001, type:int, comment:null),
            default.source_tbl(src):FieldSchema(name:col_002, type:int, comment:null),
            default.source_tbl(src):FieldSchema(name:col_003, type:int, comment:null),
            default.source_tbl(src):FieldSchema(name:BLOCK__OFFSET__INSIDE__FILE, type:bigint, comment:),
            default.source_tbl(src):FieldSchema(name:INPUT__FILE__NAME, type:string, comment:),
            default.source_tbl(src):FieldSchema(name:ROW__ID, type:struct<transactionId:bigint,bucketId:int,rowId:bigint>, comment:)
           ];
 
DependencyKey=default.target_tbl:FieldSchema(name:col_002, type:int, comment:null)
Dependency=[SCRIPT]
           [default.source_tbl(src):FieldSchema(name:col_001, type:int, comment:null),
            default.source_tbl(src):FieldSchema(name:col_002, type:int, comment:null),
            default.source_tbl(src):FieldSchema(name:col_003, type:int, comment:null),
            default.source_tbl(src):FieldSchema(name:BLOCK__OFFSET__INSIDE__FILE, type:bigint, comment:),
            default.source_tbl(src):FieldSchema(name:INPUT__FILE__NAME, type:string, comment:),
            default.source_tbl(src):FieldSchema(name:ROW__ID, type:struct<transactionId:bigint,bucketId:int,rowId:bigint>, comment:)
           ];
 
DependencyKey=default.target_tbl:FieldSchema(name:col_003, type:int, comment:null)
Dependency=[SCRIPT]
           [default.source_tbl(src):FieldSchema(name:col_001, type:int, comment:null),
            default.source_tbl(src):FieldSchema(name:col_002, type:int, comment:null),
            default.source_tbl(src):FieldSchema(name:col_003, type:int, comment:null),
            default.source_tbl(src):FieldSchema(name:BLOCK__OFFSET__INSIDE__FILE, type:bigint, comment:),
            default.source_tbl(src):FieldSchema(name:INPUT__FILE__NAME, type:string, comment:),
            default.source_tbl(src):FieldSchema(name:ROW__ID, type:struct<transactionId:bigint,bucketId:int,rowId:bigint>, comment:)
           ];

When INSERT statement doesn't include "ROW_NUMBER() OVER() AS r_num", the lineage details look correct.

Issue Links

• causes: ATLAS-2891 Incorrect column lineage: each output column has input from *all columns* of the input table