replace accesss of "fs.default.name" with FileSystem accessor methods

• Key: HADOOP-2902
• Type: Improvement
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Doug Cutting
• Reporter: Doug Cutting
• Component/s: fs
• Fix Version/s: 0.17.0

Description

HADOOP-1967 added accessor methods to set the default filesystem. We should start using them.

While doing this, we should also replace uses of "local" and "host:port" with proper URIs, e.g., "file:///" and "hdfs://host:port/". This will silence warnings about the use of old-format names.

Comments

[Doug Cutting]

Here's a nearly-complete patch. TestCrcCorruption fails with it, though.

[Doug Cutting]

The problem with TestCrcCorruption was insufficient file descriptors. This fails for me with 1024 but succeeds with 2048. Is that new?

[dhruba borthakur]

The "running out of fds" might be related to HADOOP-3051

[Doug Cutting]

New version that fixes a bug in MRCaching.java.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12378359/HADOOP-2902.patch
against trunk revision 619744.

@author +1. The patch does not contain any @author tags.

tests included +1. The patch appears to include 84 new or modified tests.

javadoc +1. The javadoc tool did not generate any warning messages.

javac +1. The applied patch does not generate any new javac compiler warnings.

release audit +1. The applied patch does not generate any new release audit warnings.

findbugs +1. The patch does not introduce any new Findbugs warnings.

core tests -1. The patch failed core unit tests.

contrib tests +1. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2023/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2023/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2023/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2023/console

This message is automatically generated.

[Doug Cutting]

MRCaching bug caused Hudson failure.

[Doug Cutting]

Try fixed version.

[Hadoop QA]

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12378396/HADOOP-2902.patch
against trunk revision 619744.

@author +1. The patch does not contain any @author tags.

tests included +1. The patch appears to include 84 new or modified tests.

javadoc +1. The javadoc tool did not generate any warning messages.

javac +1. The applied patch does not generate any new javac compiler warnings.

release audit +1. The applied patch does not generate any new release audit warnings.

findbugs +1. The patch does not introduce any new Findbugs warnings.

core tests +1. The patch passed core unit tests.

contrib tests +1. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2026/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2026/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2026/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2026/console

This message is automatically generated.

[Doug Cutting]

Can someone please review this before I commit it?

[stack]

Patch looks fine. Applied. Tests won't compile for me (r640362). Two errors of same type in same file. Here is one:

[javac] /Users/stack/Documents/checkouts/hadoop-trunk/src/test/org/apache/hadoop/mapred/TestMRServerPorts.java:103: set(java.lang.String,java.lang.String) in org.apache.hadoop.conf.Configuration cannot be applied to (java.lang.String,java.net.URI)
    [javac]       conf2.set("mapred.job.tracker",

You need to toString it?

Couple of minor items. Wouldn't put them in way of a commit.

Regards:

+      if (namenode.startsWith("hdfs://"))

...isn't there a define for "hdfs://" you could use or a method you could pass a URI to under dfs package that would tell you if its a HDFS URI? (You do "hdfs://" multiple times in TestCopyFiles. Same for the string "file:///").

Are you toStringing a URI and then creating a URI in the following code from TestCopyFiles?

+      namenode = FileSystem.getDefaultUri(conf).toString();
+      if (namenode.startsWith("hdfs://")) {
+        MyFile[] files = createFiles(URI.create(namenode), "/srcdat");

[Doug Cutting]

> Tests won't compile for me (r640362).

Yes, trunk has changed. I'll generate an updated patch.

[Doug Cutting]

Update patch for changes to trunk.

[Doug Cutting]

> You do "hdfs://" multiple times in TestCopyFiles. Same for the string "file:///"

There were lots of those before too. I think there are actually fewer now, just in different places! I don't think the situation in TestCopyFiles is made any worse.

> Are you toStringing a URI and then creating a URI in the following code from TestCopyFiles?

Yes. The namenode variable is used frequently as a String in this file. Changing it to a URI would be a substantial rewrite.

[Hadoop QA]

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12378572/HADOOP-2902.patch
against trunk revision 619744.

@author +1. The patch does not contain any @author tags.

tests included +1. The patch appears to include 84 new or modified tests.

javadoc +1. The javadoc tool did not generate any warning messages.

javac +1. The applied patch does not generate any new javac compiler warnings.

release audit +1. The applied patch does not generate any new release audit warnings.

findbugs +1. The patch does not introduce any new Findbugs warnings.

core tests +1. The patch passed core unit tests.

contrib tests +1. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2052/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2052/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2052/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2052/console

This message is automatically generated.

[Doug Cutting]

I committed this.

[Hudson]

Integrated in Hadoop-trunk #443 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/443/)

[Nigel Daley]

This patch has caused org.apache.hadoop.mapred.TestMiniMRLocalFS.testWithLocal to fail since it was committed. See HADOOP-3157.