[SPARK-18960] Avoid double reading file which is being copied.

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.2
• Fix Version/s: 2.2.0
• Component/s: SQL, Structured Streaming
• Assignee: Genmao Yu
• Reporter: Genmao Yu
• Created: 21/Dec/16 06:28
• Updated: 28/Dec/16 10:43
• Resolved: 28/Dec/16 10:42

Description

In HDFS, when we copy a file into target directory, there will a temporary .COPY file for a period of time. The duration depends on file size. If we do not skip this file, we will may read the same data for two times.

Issue Links

• links to: [Github] Pull Request #16370 (uncleGen)