We should discuss just community version, not specific distribution.
But what you say seems to affect Hadoop-2 and trunc.

Correct, the above holds in the community version; before submitting this jira I checked the (apache) trunk.

OK. I think we should change Affects Version/s to trunc.

I succeeded to reproduce in branch-2.1-beta.
I saw a temporary file with prefix ".COPYING" during putting a file into HDFS.
As you say, it causes something bad that we run MapReduce jobs when there are .COPYING files in the directory where MapReduce jobs use for input path.

Stan, I think we should modify FsShell to create a file assigned the name with underscore prefix for a file being created so that FileInputFormat can ignore the file rather than modify FileInputFormat to handle a .COPYING suffix file as a hidden file.
It's just HDFS matter and I think the specification change shouldn't affects MapReduce.
How do you think?

I agree. It appears this change is confined to FsShell and nothing else. Do we know why this particular file naming convention was chosen or was it just an oversight?

I found the jira where the code which ".COPYING" temporary file being created is added.
https://issues.apache.org/jira/browse/HADOOP-7771
In that jira, they discussed NPE problem when using copyToLocal and the reason why the ".COPYING" is created is to copy file persistently.
So, I think the temporary file is not necessarily assigned the name with ".COPYING" suffix.

I don't think giving this responsibility to FileInputFormat is a good idea. FileInputFormat already provides extensibility to add new filters using "mapred.input.pathFilter.class" configuration. If the user want to filter some specific files from the input dir for some Jobs they can achieve the same using the current behavior.

Kousuke,

HADOOP-7771 is a very interesting find! Based on my reading of the history, the naming convention uses the suffix explicitly to make the corresponding temporary (staging) file non-hidden. Indeed, the last comment in that jira is a question by Daryn w.r.t. whether or not partially copied files should be visible; there are no follow-ups to the question but the jira is closed. Since the decision to have temporary files visible seems to be arbitrary, I propose that we fix FsShell by making them hidden. Otherwise, this notion of partially loaded files needs to be lifted to the level of FileInputFormat---jobs should not be failing under normal conditions.

As Devaraj said, we can use "mapred.input.pathFilter.class" but, as far as I know, the name of the temporary file is undocumented and I think changes of the specification or implementation of HDFS should not affect users who have ever used HDFS.
So, I think we should consider the name of the temporary file. It may good that the name of the temporary file starts with "." or "_".

May I recreate this jira as HDFS issue?

I have no objections, but, perhaps we can get committers in HADOOP-7771 to comment on whether or not they think this belongs to HDFS?

Thanks Stan. I have asked in HADOOP-7771.

Why are you running a Map/Reduce job with input from a directory that has not finished being copied? MR was not designed to run on data that is changing underneath it. When the job is done how do you know which of the input files were actually used to produce the output? This issue existed prior to 2.0 but was even worse without the .COPYING suffix. In those cases the files were opened in place and data started to be copied into them. You may have only even gotten a part of the file in your MR job, not all of it. The file could have disappeared out from under the MR job if an error occurred.

This is not behavior that I want to make a common park of Map/Reduce. If you want to do this and you know the risks then you can filter .COPYING files out of your list of input files to the MR job. But I don't want the framework to do it automatically for everyone.

I am happy to hear arguments as to why this is really necessary, but I would rather have my job fail then have the job give me partial/inconsistent results.

Robert, what is the intended operational meaning of 'hdfs fs put local dst'? Is it not that file denoted by local is "atomically" transferred into dst? If that's the case, then I'd argue that it's broken--from MR perspective the transfer is not truly atomic since files with the suffix .COPYING are visible.

As I've indicated above, we have jobs which execute as soon as new data is available for that (hdfs) partition. The external scheduler knows when new data has finished loading, namely when all pending hdfs 'put' operations complete.
(Think of it as a special type of job in the sense that it runs many times per hour, every time processing a superset of the input files.)

Your claim that MR was not designed to run on data that is changing underneath it seems rather putative. What is wrong with the above approach assuming that the intended semantics of 'put' is atomic transfer without (MR) observable side effect of .COPYING? (In other words, if MR is oblivious to .COPYING, then data it not changing underneath it.)