[HIVE-25025] Distcp In MoveTask may cause stats info lost

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: Hive
• Labels: patch pull-request-available
• Flags: Patch, Important
• Assignee: WangHualei
• Reporter: WangHualei
• Created: 19/Apr/21 07:17
• Updated: 26/Jun/21 00:09
• Resolved: 19/Apr/21 08:13

Description

after set  Run as end user instead of Hive user , when execute insert overwrite , In MoveTask ,if source byte > HIVE_EXEC_COPYFILE_MAXSIZE  and source file count> HIVE_EXEC_COPYFILE_MAXNUMFILES , HIve will use distcp method, it may cause tmp stats file lost. 

example:

set hive.exec.copyfile.maxsize=0;
set hive.exec.copyfile.maxnumfiles=0;

insert overwrite table abc_new select * from abc;

select count(1) from abc_new ;

select * from abc_new ;

then the  count(1) result will be 0, but select * will display real data, because stats info lost.

 

Issue Links

• links to: GitHub Pull Request #2187