Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Duplicate
-
4.0.0
Description
HMS translation layer convert the table to external based on the transactional property set to false but MoveTask does not entertain the translated table location and move the data to the managed table location;
steps to repro:
create table nontxnal TBLPROPERTIES ('transactional'='false') as select * from abc;
select query on table return nothing t but the source table has data in it.
select * from nontxnal; +--------------+ | nontxnal.id | +--------------+ +--------------+
--show create table
CREATE EXTERNAL TABLE `nontxnal`( | | `id` int) | | ROW FORMAT SERDE | | 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' | | STORED AS INPUTFORMAT | | 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' | | OUTPUTFORMAT | | 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' | | LOCATION | | 'hdfs://hostname:8020/warehouse/tablespace/external/hive/nontxnal' | | TBLPROPERTIES ( | | 'TRANSLATED_TO_EXTERNAL'='TRUE', | | 'bucketing_version'='2', | | 'external.table.purge'='TRUE', | | 'transient_lastDdlTime'='1596215634') |
table data is moved to the managed location:
```
dfs -ls -R hdfs://hostname:8020/warehouse/tablespace/managed/hive/nontxnal
. . . . . . . . . . . . . . . . . . . . . . .> ;
----------------------------------------------------
DFS Output |
----------------------------------------------------
|
----------------------------------------------------
```
The problem seems to be here
isExternal evaluates to false since the statement is missing external
https://github.com/apache/hive/blob/d4bfd2ea1ee797f53227f447749cbc97803cd5dc/ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java#L446
and location return to the managed location
https://github.com/apache/hive/blob/d4bfd2ea1ee797f53227f447749cbc97803cd5dc/ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java#L455
Attachments
Issue Links
- duplicates
-
HIVE-24625 CTAS with TBLPROPERTIES ('transactional'='false') loads data into incorrect directory
- Closed
- links to