[MAPREDUCE-5247] FileInputFormat should filter files with '._COPYING_' sufix

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Assignee: Unassigned
• Reporter: Stan Rosenberg
• Created: 14/May/13 15:36
• Updated: 19/Jul/13 16:59
• Resolved: 19/Jul/13 15:02

Description

FsShell copy/put creates staging files with '.COPYING' suffix. These files should be considered hidden by FileInputFormat. (A simple fix is to add the following conjunct to the existing hiddenFilter:

!name.endsWith("._COPYING_")

After upgrading to CDH 4.2.0 we encountered this bug. We have a legacy data loader which uses 'hadoop fs -put' to load data into hourly partitions. We also have intra-hourly jobs which are scheduled to execute several times per hour using the same hourly partition as input. Thus, as the new data is continuously loaded, these staging files (i.e., .COPYING) are breaking our jobs (since when copy/put completes staging files are moved).

As a workaround, we've defined a custom input path filter and loaded it with "mapred.input.pathFilter.class".

Issue Links

• relates to: HADOOP-9750 '._COPYING_' sufix temp file could prevent from running MapReduce job