Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.18.0
    • Component/s: util
    • Labels:
      None

      Description

      This issue aims at providing a handler to resolve DFS URLs ("hdfs://host:port/file/to/path"), so that such URLs can be read using the URL API (mainly InputStream url.openStream()).

      This allows the use of a URLClassLoader which would serve classes directly from the DFS.

      1. 3074-20080324a.patch
        7 kB
        Christophe Taton
      2. 3074-20080324b.patch
        9 kB
        Christophe Taton
      3. 3074-20080325a.patch
        11 kB
        Christophe Taton
      4. 3074-20080406a.patch
        11 kB
        Christophe Taton
      5. 3074-20080412a.patch
        11 kB
        Christophe Taton
      6. 3074-20080412b.patch
        12 kB
        Christophe Taton
      7. 3074-20080414a.patch
        12 kB
        Christophe Taton

        Activity

        Hide
        kryzthov Christophe Taton added a comment -

        Here is a first proposal. The handler has been put in package org.apache.hadoop.util.protocols.hdfs. Thus you can register it in the JVM as in: java -Djava.protocol.handler.pkgs=org.apache.hadoop.util.protocols <MainClass>.

        Show
        kryzthov Christophe Taton added a comment - Here is a first proposal. The handler has been put in package org.apache.hadoop.util.protocols.hdfs . Thus you can register it in the JVM as in: java -Djava.protocol.handler.pkgs=org.apache.hadoop.util.protocols <MainClass> .
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12378434/3074-20080322a.patch
        against trunk revision 619744.

        @author +1. The patch does not contain any @author tags.

        tests included +1. The patch appears to include 3 new or modified tests.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new javac compiler warnings.

        release audit -1. The applied patch generated 197 release audit warnings (more than the trunk's current 195 warnings).

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests +1. The patch passed core unit tests.

        contrib tests +1. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2034/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2034/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2034/artifact/trunk/build/test/checkstyle-errors.html
        Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2034/artifact/trunk/current/releaseAuditDiffWarnings.txt
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2034/console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12378434/3074-20080322a.patch against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 3 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit -1. The applied patch generated 197 release audit warnings (more than the trunk's current 195 warnings). findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2034/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2034/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2034/artifact/trunk/build/test/checkstyle-errors.html Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2034/artifact/trunk/current/releaseAuditDiffWarnings.txt Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2034/console This message is automatically generated.
        Hide
        kryzthov Christophe Taton added a comment -

        Fixing missing license headers...

        Show
        kryzthov Christophe Taton added a comment - Fixing missing license headers...
        Hide
        hadoopqa Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12378477/3074-20080324a.patch
        against trunk revision 619744.

        @author +1. The patch does not contain any @author tags.

        tests included +1. The patch appears to include 3 new or modified tests.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new javac compiler warnings.

        release audit +1. The applied patch does not generate any new release audit warnings.

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests +1. The patch passed core unit tests.

        contrib tests +1. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2035/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2035/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2035/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2035/console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12378477/3074-20080324a.patch against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 3 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2035/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2035/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2035/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2035/console This message is automatically generated.
        Hide
        cutting Doug Cutting added a comment -

        This need not be specific to HDFS, but could be generic to any FileSystem implementation, no? The FileSystem API subsumes Java's URL connection API. We could even provide a generic URLStreamHandlerFactory.

        http://java.sun.com/javase/6/docs/api/java/net/URL.html#setURLStreamHandlerFactory(java.net.URLStreamHandlerFactory)

        Note that we might special-case the "file:" protocol, since the JVM already includes an handler for that.

        Show
        cutting Doug Cutting added a comment - This need not be specific to HDFS, but could be generic to any FileSystem implementation, no? The FileSystem API subsumes Java's URL connection API. We could even provide a generic URLStreamHandlerFactory. http://java.sun.com/javase/6/docs/api/java/net/URL.html#setURLStreamHandlerFactory(java.net.URLStreamHandlerFactory ) Note that we might special-case the "file:" protocol, since the JVM already includes an handler for that.
        Hide
        kryzthov Christophe Taton added a comment -

        The patch 3074-20080324b.patch is not DFS specific anymore, as suggested by Doug.

        Show
        kryzthov Christophe Taton added a comment - The patch 3074-20080324b.patch is not DFS specific anymore, as suggested by Doug.
        Hide
        cutting Doug Cutting added a comment -

        +1 This looks good to me.

        Show
        cutting Doug Cutting added a comment - +1 This looks good to me.
        Hide
        rangadi Raghu Angadi added a comment -

        Looks good.

        minor:

        Do you need to use FSDataInputStream or FSDataOutputStream in the test or in the implementation. Looks like just normal InputStream etc are enough.

        Show
        rangadi Raghu Angadi added a comment - Looks good. minor: Do you need to use FSDataInputStream or FSDataOutputStream in the test or in the implementation. Looks like just normal InputStream etc are enough.
        Hide
        kryzthov Christophe Taton added a comment -

        I dug concerning file:// URLs and the previous patch does not work for these URLs.
        This happens because of the late loading of Configuration resource files (which itself relies on URLs).
        The new patch forces the load of such resource files before setting the new handler factory, thus allowing it to handle file:// URLs too.

        The code has also been cleaned (FSDataInput/OutputStream replaced with Input/OutputStream).

        Show
        kryzthov Christophe Taton added a comment - I dug concerning file:// URLs and the previous patch does not work for these URLs. This happens because of the late loading of Configuration resource files (which itself relies on URLs). The new patch forces the load of such resource files before setting the new handler factory, thus allowing it to handle file:// URLs too. The code has also been cleaned (FSDataInput/OutputStream replaced with Input/OutputStream).
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12378573/3074-20080325a.patch
        against trunk revision 619744.

        @author +1. The patch does not contain any @author tags.

        tests included +1. The patch appears to include 3 new or modified tests.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new javac compiler warnings.

        release audit +1. The applied patch does not generate any new release audit warnings.

        findbugs -1. The patch appears to cause Findbugs to fail.

        core tests -1. The patch failed core unit tests.

        contrib tests -1. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2055/testReport/
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2055/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2055/console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12378573/3074-20080325a.patch against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 3 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs -1. The patch appears to cause Findbugs to fail. core tests -1. The patch failed core unit tests. contrib tests -1. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2055/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2055/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2055/console This message is automatically generated.
        Hide
        kryzthov Christophe Taton added a comment -

        New patch to conform to findbugs

        Show
        kryzthov Christophe Taton added a comment - New patch to conform to findbugs
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12379511/3074-20080406a.patch
        against trunk revision 645773.

        @author +1. The patch does not contain any @author tags.

        tests included +1. The patch appears to include 3 new or modified tests.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new javac compiler warnings.

        release audit +1. The applied patch does not generate any new release audit warnings.

        findbugs -1. The patch appears to cause Findbugs to fail.

        core tests -1. The patch failed core unit tests.

        contrib tests -1. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2213/testReport/
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2213/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2213/console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12379511/3074-20080406a.patch against trunk revision 645773. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 3 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs -1. The patch appears to cause Findbugs to fail. core tests -1. The patch failed core unit tests. contrib tests -1. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2213/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2213/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2213/console This message is automatically generated.
        Hide
        kryzthov Christophe Taton added a comment -

        Fixing compilation failure due to the constructor new IOException(Throwable) which does not exist before Java6.

        Show
        kryzthov Christophe Taton added a comment - Fixing compilation failure due to the constructor new IOException(Throwable) which does not exist before Java6.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12379973/3074-20080412a.patch
        against trunk revision 645773.

        @author +1. The patch does not contain any @author tags.

        tests included +1. The patch appears to include 3 new or modified tests.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new javac compiler warnings.

        release audit +1. The applied patch does not generate any new release audit warnings.

        findbugs -1. The patch appears to introduce 3 new Findbugs warnings.

        core tests +1. The patch passed core unit tests.

        contrib tests +1. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2216/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2216/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2216/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2216/console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12379973/3074-20080412a.patch against trunk revision 645773. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 3 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs -1. The patch appears to introduce 3 new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2216/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2216/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2216/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2216/console This message is automatically generated.
        Hide
        kryzthov Christophe Taton added a comment -

        Fixing all findbugs warnings.

        Show
        kryzthov Christophe Taton added a comment - Fixing all findbugs warnings.
        Hide
        hadoopqa Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12379976/3074-20080412b.patch
        against trunk revision 645773.

        @author +1. The patch does not contain any @author tags.

        tests included +1. The patch appears to include 3 new or modified tests.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new javac compiler warnings.

        release audit +1. The applied patch does not generate any new release audit warnings.

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests +1. The patch passed core unit tests.

        contrib tests +1. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2218/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2218/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2218/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2218/console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12379976/3074-20080412b.patch against trunk revision 645773. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 3 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2218/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2218/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2218/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2218/console This message is automatically generated.
        Hide
        cutting Doug Cutting added a comment -

        I would prefer this were in the fs package, not util. In general, we should aim to put things in the most-specific package possible, otherwise, everything winds up in util.

        Also, I would prefer the classes were named FsUrlStreamHandler and FsUrlStreamHandlerFactory. Acronyms are easier to read when capitalized, not all-caps, in Java names.

        Show
        cutting Doug Cutting added a comment - I would prefer this were in the fs package, not util. In general, we should aim to put things in the most-specific package possible, otherwise, everything winds up in util. Also, I would prefer the classes were named FsUrlStreamHandler and FsUrlStreamHandlerFactory. Acronyms are easier to read when capitalized, not all-caps, in Java names.
        Hide
        kryzthov Christophe Taton added a comment -

        Here you are.

        Show
        kryzthov Christophe Taton added a comment - Here you are.
        Hide
        cutting Doug Cutting added a comment -

        +1

        Show
        cutting Doug Cutting added a comment - +1
        Hide
        hadoopqa Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12380090/3074-20080414a.patch
        against trunk revision 645773.

        @author +1. The patch does not contain any @author tags.

        tests included +1. The patch appears to include 3 new or modified tests.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new javac compiler warnings.

        release audit +1. The applied patch does not generate any new release audit warnings.

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests +1. The patch passed core unit tests.

        contrib tests +1. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2229/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2229/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2229/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2229/console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12380090/3074-20080414a.patch against trunk revision 645773. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 3 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2229/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2229/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2229/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2229/console This message is automatically generated.
        Hide
        kryzthov Christophe Taton added a comment -

        I just committed this. Thanks!

        Show
        kryzthov Christophe Taton added a comment - I just committed this. Thanks!
        Hide
        hudson Hudson added a comment -
        Show
        hudson Hudson added a comment - Integrated in Hadoop-trunk #461 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/461/ )

          People

          • Assignee:
            kryzthov Christophe Taton
            Reporter:
            kryzthov Christophe Taton
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development