Remove deprecated methods introduced in changes to validating input paths (HADOOP-3095)

• Key: HADOOP-3664
• Type: Improvement
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Tom White
• Reporter: Tom White
• Fix Version/s: 0.19.0

Description

Remove FileInputFormat#listPaths, FileInputFormat#validateInput, FileInputFormat#getSplitsForPaths, FileSystem#getFileBlockLocations(Path, long, long)

Comments

[Tom White]

I'm wondering whether we want to remove FileInputFormat#validateInput - is it needed to do job validation (e.g. HADOOP-1683), or would that be better done elsewhere?

Also, FileSystem#getFileBlockLocations(Path, long, long) is actually a convenience method for FileSystem#getFileBlockLocations(FileStatus, long, long), so there's not much harm in leaving it there. It's used quite widely. FileSystem implementations only need to implement the latter signature, so it's not a burden on them.

If we decide not to remove either or both of these methods then we would just undeprecate them.

[Doug Cutting]

> I'm wondering whether we want to remove FileInputFormat#validateInput

The original need for this method was to be able to check inputs prior to submitting the job, since splits were then generated at the jobtracker. Now that splits are generated in the jobclient, there's not much point to keeping this method: input validation can be done when splits are generated, still providing rapid feedback.

> FileSystem#getFileBlockLocations(Path, long, long) is actually a convenience method

That's also true of #listPaths(), no? I don't have a strong feeling here. Keeping a convenience method isn't bad, but, we also want to encourage folks to write efficient code that does not call #getStatus too frequently.

[Tom White]

Here's a patch that removes all of these deprecated methods.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12387810/hadoop-3664.patch
against trunk revision 683671.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 63 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3039/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3039/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3039/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3039/console

This message is automatically generated.

[Owen O'Malley]

I just committed this. Thanks, Tom!

[Hudson]

Integrated in Hadoop-trunk #581 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/581/)