[HDFS-13660] DistCp job fails when new data is appended in the file while the distCp copy job is running

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 3.3.0, 3.1.4, 3.2.2
• Component/s: distcp
• Assignee: Mukund Thakur
• Reporter: Mukund Thakur
• Created: 07/Jun/18 10:09
• Updated: 28/Feb/20 00:38
• Resolved: 24/Sep/19 10:27

Description

Steps to reproduce: 

Suppose distcp MR job is copying the file /tmp/web_returns_merged/data-m-002 and 

we append some more data to this file using command 

hadoop fs -appendToFile xaa  /tmp/web_returns_merged/data-m-002

the job fails with exception 

Mismatch in length of source:hdfs://mycluster0/tmp/web_returns_merged/data-m-002 and target.

Attached the logs.

Issue Links

• links to: GitHub Pull Request #1404