[IMPALA-9096] Create external table ddls should send column lineages.

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: Frontend
• Epic Color: ghx-label-9
• Assignee: Unassigned
• Reporter: Anurag Mantripragada
• Created: 25/Oct/19 22:51
• Updated: 13/Feb/20 00:02

Description

Create external table with specified columns should create column lineages for tools like Altas to consume.

 

For example:

create EXTERNAL TABLE IF NOT EXISTS friday_ext6
(STUD_ID int,
DEPT_ID int,
NAME string
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ‘,’
STORED AS TEXTFILE;

Currently we send a lineage like:

 {
 "queryText":"create EXTERNAL TABLE IF NOT EXISTS friday_ext5 (STUD_ID int, DEPT_ID    int, NAME string ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘,’ STORED AS TEXTFILE LOCATION ‘/warehouse/tablespace/external/hive/testdb.db/friday_ext5’",
 "queryId":"4b471ac0ca2b0f93:029db79c00000000",
 "hash":"867fae20bc6c8254c05774cc923a99fa",
 "user":"admin",
 "timestamp":1572028716,
 "endTime":1572028716,
 "edges":[],
 "vertices":[],
 "tableLocation":"hdfs://sid-cdp-2-1.gce.cloudera.com:8020/warehouse/tablespace/external/hive/testdb.db/friday_ext"
}
 

Atlas needs fully qualified table name to create lineage.