Hadoop Common
  1. Hadoop Common
  2. HADOOP-5635

distributed cache doesn't work with other distributed file systems

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.20.0
    • Fix Version/s: 0.21.0
    • Component/s: filecache
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently the DistributedCache does a check to see if the file to be included is an HDFS URI. If the URI isn't in HDFS, it returns the default filesystem. This prevents using other distributed file systems – such as s3, s3n, or kfs – with distributed cache. When a user tries to use one of those filesystems, it reports an error that it can't find the path in HDFS.

      1. HADOOP-5635.patch
        6 kB
        Andrew Hitchcock
      2. fix-distributed-cache.patch
        1 kB
        Andrew Hitchcock

        Activity

        Hide
        Andrew Hitchcock added a comment -

        This patch removes the check for HDFS and lets any file system through. The onus is on the user to ensure that the file system is globally available on all nodes.

        Show
        Andrew Hitchcock added a comment - This patch removes the check for HDFS and lets any file system through. The onus is on the user to ensure that the file system is globally available on all nodes.
        Hide
        Craig Macdonald added a comment -

        Wouldn't it be generally better if Hadoop was configured with a list of shared file systems. Then, when the administrator permitted, users could use shared NFS filesystems as sources and targets for map reduce jobs. E.g., in our setup, /local/ and /users/ are shared to all nodes. If we wanted to do a quick map reduce test on stuff storied in /local/ we would have to copy to the DFS, when it would be OK to run as is.

        <name>fs.shared.filesystems</name>
        <value>hdfs://,file://users/,file://local/ </value>
        
        Show
        Craig Macdonald added a comment - Wouldn't it be generally better if Hadoop was configured with a list of shared file systems. Then, when the administrator permitted, users could use shared NFS filesystems as sources and targets for map reduce jobs. E.g., in our setup, /local/ and /users/ are shared to all nodes. If we wanted to do a quick map reduce test on stuff storied in /local/ we would have to copy to the DFS, when it would be OK to run as is. <name>fs.shared.filesystems</name> <value>hdfs://,file://users/,file://local/ </value>
        Hide
        Andrew Hitchcock added a comment -

        It sounds like what you want is a new featue whereas this patch is just to fix a bug.

        Currently the behavior is not right. If a user specifies a non-HDFS URI for distributed cache then the job will fail because the tasks look for the file in HDFS. This patch fixes that for cases when the user specifies a URI to another distributed file system. With the patch, if a user specifies KFS or S3N (and the file system is properly configured) then the job will succeed. The behavior for specifying a URI not accessible on every machine remains unchanged: the job will fail as tasks are unable to reach the URI.

        I think a feature for administrators to restrict distributed cache access to certain file systems should be a new Jira.

        Show
        Andrew Hitchcock added a comment - It sounds like what you want is a new featue whereas this patch is just to fix a bug. Currently the behavior is not right. If a user specifies a non-HDFS URI for distributed cache then the job will fail because the tasks look for the file in HDFS. This patch fixes that for cases when the user specifies a URI to another distributed file system. With the patch, if a user specifies KFS or S3N (and the file system is properly configured) then the job will succeed. The behavior for specifying a URI not accessible on every machine remains unchanged: the job will fail as tasks are unable to reach the URI. I think a feature for administrators to restrict distributed cache access to certain file systems should be a new Jira.
        Hide
        Tom White added a comment -

        Andrew,

        This looks like a good change to me. Have you thought how to write a unit test for this?

        Also, the documentation in DistributedCache should be updated to remove HDFS assumptions.

        Show
        Tom White added a comment - Andrew, This looks like a good change to me. Have you thought how to write a unit test for this? Also, the documentation in DistributedCache should be updated to remove HDFS assumptions.
        Hide
        Andrew Hitchcock added a comment -

        I've updated the patch. It includes a unit test, fixes the error message in StreamJob, and updates some Javadocs that I had previously missed.

        Show
        Andrew Hitchcock added a comment - I've updated the patch. It includes a unit test, fixes the error message in StreamJob, and updates some Javadocs that I had previously missed.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12408837/HADOOP-5635.patch
        against trunk revision 778388.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/403/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/403/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/403/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/403/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12408837/HADOOP-5635.patch against trunk revision 778388. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/403/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/403/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/403/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/403/console This message is automatically generated.
        Hide
        Andrew Hitchcock added a comment -

        The failing tests seem unrelated.

        Show
        Andrew Hitchcock added a comment - The failing tests seem unrelated.
        Hide
        Tom White added a comment -

        I've just committed this. Thanks Andrew!

        Show
        Tom White added a comment - I've just committed this. Thanks Andrew!
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #863 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/863/ )

          People

          • Assignee:
            Andrew Hitchcock
            Reporter:
            Andrew Hitchcock
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development