JobConf should have a setInputPathFilter method

• Key: HADOOP-2055
• Type: New Feature
• Status: Closed
• Resolution: Fixed
• Priority: Minor
• Assignee: Alejandro Abdelnur
• Reporter: Alejandro Abdelnur
• Fix Version/s: 0.17.0

Description

It should be possible to set a PathFilter for the input to avoid taking certain files as input data within the input directories.

Comments

[Owen O'Malley]

This should be a static method on the FileInputFormat instead of JobConf, since it won't affect the framework, but only the FileInputFormat's behavior.

[Owen O'Malley]

The method should probably also have a getter and most of them look like:

public static void setInputPathFilter(JobConf job, PathFilter filter);
public static PathFilter getInputPathFilter(JobConf job);

[Alejandro Abdelnur]

Having a static method on the FileInputFormat it would make difficult for an application that dispatches hadoop jobs (ie a webapp) to set filters on per job basis.

IMO, it should be configurable at job level.

[Doug Cutting]

> IMO, it should be configurable at job level.

Please look more closely at the static methods Owen suggested. The job is a parameter.

[eric baldeschwieler]

we support globing in input paths now. Doesn't that address this need?

IE *.foo

[Alejandro Abdelnur]

Owen, Doug, got the static methos thing, that would work.

Eric, using wildcards would not work as it allows you to tell what you want to include, but now what you don't want to include.

For example, if I have some files like the CRC files (to track other type of information) and I would like to skip them.

[Alejandro Abdelnur]

I've figured out (IMO) a cleaner way of implementing this feature:

Adding the following 2 instance methods to the JobConf:

• void setInputPathFilter(class<? extends PathFilter> pathFilter);
• InputPathFilter getInputPathFilter();

Modifying the FileInputFormat's listPaths() method to apply the hiddenFileFilter and (if set) the filter set in the jobconf.

And still globbing works for regex inclusion, even if a path filter is set.

By being able to specify a custom PathFilter it will be possible to create more complex filters such as exclusion ones and doing selections not possible to be done via regex.

[Hadoop QA]

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12378554/patch2055.txt
against trunk revision 619744.

@author +1. The patch does not contain any @author tags.

tests included +1. The patch appears to include 3 new or modified tests.

javadoc +1. The javadoc tool did not generate any warning messages.

javac +1. The applied patch does not generate any new javac compiler warnings.

release audit +1. The applied patch does not generate any new release audit warnings.

findbugs +1. The patch does not introduce any new Findbugs warnings.

core tests +1. The patch passed core unit tests.

contrib tests +1. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2048/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2048/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2048/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2048/console

This message is automatically generated.

[Alejandro Abdelnur]

refactored patch to Owen's suggestion as the functionality is specific to File InputFormats.

[Hadoop QA]

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12378623/patch2055.txt
against trunk revision 619744.

@author +1. The patch does not contain any @author tags.

tests included +1. The patch appears to include 3 new or modified tests.

javadoc +1. The javadoc tool did not generate any warning messages.

javac +1. The applied patch does not generate any new javac compiler warnings.

release audit +1. The applied patch does not generate any new release audit warnings.

findbugs +1. The patch does not introduce any new Findbugs warnings.

core tests +1. The patch passed core unit tests.

contrib tests +1. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2061/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2061/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2061/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2061/console

This message is automatically generated.

[Devaraj Das]

I just committed this. Thanks, Alejandro!

[Hudson]

Integrated in Hadoop-trunk #445 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/445/)