Hadoop Common
  1. Hadoop Common
  2. HADOOP-7549

Use JDK ServiceLoader mechanism to find FileSystem implementations

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 2.0.0-alpha
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently configuring FileSystem implementations must be done by declaring the FileSystem class in the Hadoop configuration files (core-default.xml, ...).

      Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the JAR file with the additional FileSystem implementation would suffice.

      This is similar to what is being proposed for compression codecs (HADOOP-7350).

      1. HADOOP-7549v3.patch
        20 kB
        Alejandro Abdelnur
      2. HADOOP-7549v2.patch
        22 kB
        Alejandro Abdelnur
      3. HADOOP-7549v1.patch
        18 kB
        Alejandro Abdelnur

        Issue Links

          Activity

          Hide
          Alejandro Abdelnur added a comment -

          The current loading mechanism (from Hadoop configuration files) would be preserved and have precedence (although it would be deprecated).

          A new method getScheme() in the FileSystem class would publish the scheme handled by the implementation.

          The base FileSystem class will implement this method throwing an UnsupportedOperationException.

          All FileSystem implementations in Hadoop will be retrofitted to implement this method declaring their scheme and they will be registered as services.

          Custom FileSystem implementations not part of Hadoop will not break if they don't implement the getScheme() method. The will continue to work as before, (they just won't support the service loading). In a future release they could be retrofitted to use the service mechanism (if their authors choose to).

          Because implementations registered in the configuration have precedence over implementations registered as services, modified implementations of Hadoop provided implementations will still override the service definitions.

          Show
          Alejandro Abdelnur added a comment - The current loading mechanism (from Hadoop configuration files) would be preserved and have precedence (although it would be deprecated). A new method getScheme() in the FileSystem class would publish the scheme handled by the implementation. The base FileSystem class will implement this method throwing an UnsupportedOperationException . All FileSystem implementations in Hadoop will be retrofitted to implement this method declaring their scheme and they will be registered as services. Custom FileSystem implementations not part of Hadoop will not break if they don't implement the getScheme() method. The will continue to work as before, (they just won't support the service loading). In a future release they could be retrofitted to use the service mechanism (if their authors choose to). Because implementations registered in the configuration have precedence over implementations registered as services, modified implementations of Hadoop provided implementations will still override the service definitions.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12490690/HADOOP-7549v1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/46//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12490690/HADOOP-7549v1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/46//console This message is automatically generated.
          Hide
          Suresh Srinivas added a comment -

          How does this handle two file system implementations with same scheme? Currently it is handled by picking the one in the configuration. Also, there might be code in mapreduce, that could be making use of config based mechanism for choosing file systems.

          Show
          Suresh Srinivas added a comment - How does this handle two file system implementations with same scheme? Currently it is handled by picking the one in the configuration. Also, there might be code in mapreduce, that could be making use of config based mechanism for choosing file systems.
          Hide
          Alejandro Abdelnur added a comment -

          Suresh,

          Configuration defined filesystems have priority over service defined file systems.

          Maybe we shouldn't say config loading is deprecated, and we should remove the warning I'm putting there.

          Regarding mapreduce, would you point me to the place where that happens?

          Thanks.

          Alejandro

          Show
          Alejandro Abdelnur added a comment - Suresh, Configuration defined filesystems have priority over service defined file systems. Maybe we shouldn't say config loading is deprecated, and we should remove the warning I'm putting there. Regarding mapreduce, would you point me to the place where that happens? Thanks. Alejandro
          Hide
          Suresh Srinivas added a comment -

          Maybe we shouldn't say config loading is deprecated, and we should remove the warning I'm putting there.

          That is what I was getting at

          Regarding mapreduce, would you point me to the place where that happens?

          grep fs.*impl in java files

          Given that we cannot deprecate the config based mechanism, which in addition to the mechanism you are proposing, allows changing FileSystem implementation during run time, why do we need this scheme?

          Show
          Suresh Srinivas added a comment - Maybe we shouldn't say config loading is deprecated, and we should remove the warning I'm putting there. That is what I was getting at Regarding mapreduce, would you point me to the place where that happens? grep fs.*impl in java files Given that we cannot deprecate the config based mechanism, which in addition to the mechanism you are proposing, allows changing FileSystem implementation during run time, why do we need this scheme?
          Hide
          Alejandro Abdelnur added a comment -

          From a user perpective, just adding a JAR with a new FileSystem is enough to use it, no need to set configurations in conf/ or in their APP.

          Regarding the direct use use of 'fs.*.impl', I want to take a look to see why is that needed.

          Show
          Alejandro Abdelnur added a comment - From a user perpective, just adding a JAR with a new FileSystem is enough to use it, no need to set configurations in conf/ or in their APP. Regarding the direct use use of 'fs.*.impl', I want to take a look to see why is that needed.
          Hide
          Suresh Srinivas added a comment -

          > From a user perpective, just adding a JAR with a new FileSystem is enough to use it, no need to set configurations in conf/ or in their APP.
          Typically configuration is preset in core-default.xml and users do not need to change it.

          Only frameworks such as mapreduce and raid change it.

          Show
          Suresh Srinivas added a comment - > From a user perpective, just adding a JAR with a new FileSystem is enough to use it, no need to set configurations in conf/ or in their APP. Typically configuration is preset in core-default.xml and users do not need to change it. Only frameworks such as mapreduce and raid change it.
          Hide
          Alejandro Abdelnur added a comment -

          not really, If you are using a custom FS implementation (ie HOOP) currently it has to be added in the client side.

          Show
          Alejandro Abdelnur added a comment - not really, If you are using a custom FS implementation (ie HOOP) currently it has to be added in the client side.
          Hide
          Suresh Srinivas added a comment -

          My main concern is - multiple implementations could be using the same scheme names (example RawLocalFileSystem, LocalFileSystem etc). I am not sure which one wins in that case. Also some one adding a jar to the path, with say an experimental file system (say HDFS) could override the DistributedFileSystem and not even know about it.

          We should consider adding HOOP to the config file.

          Show
          Suresh Srinivas added a comment - My main concern is - multiple implementations could be using the same scheme names (example RawLocalFileSystem, LocalFileSystem etc). I am not sure which one wins in that case. Also some one adding a jar to the path, with say an experimental file system (say HDFS) could override the DistributedFileSystem and not even know about it. We should consider adding HOOP to the config file.
          Hide
          Alejandro Abdelnur added a comment -

          Regarding the first concern, conf wins over service. So there is always an override.

          Regarding the second concern, you are right in a sense. That is similar to adding a JAR with duplicated classes.

          Adding Hoop to the config file would be a point solution for Hoop, but not for other FS impls.

          Show
          Alejandro Abdelnur added a comment - Regarding the first concern, conf wins over service. So there is always an override. Regarding the second concern, you are right in a sense. That is similar to adding a JAR with duplicated classes. Adding Hoop to the config file would be a point solution for Hoop, but not for other FS impls.
          Hide
          Owen O'Malley added a comment -

          I think we need to move this direction. The cases where we have multiple file systems with the same "protocol" are very problematic, especially as we move to the FileContext API.

          We should probably rename:
          file -> raw local
          checksum -> local

          are there any other overlapping pairs?

          Show
          Owen O'Malley added a comment - I think we need to move this direction. The cases where we have multiple file systems with the same "protocol" are very problematic, especially as we move to the FileContext API. We should probably rename: file -> raw local checksum -> local are there any other overlapping pairs?
          Hide
          Alejandro Abdelnur added a comment -

          @Owen, are you suggesting we should de-overlap schema names? If so, agreed.

          Some how related, it was mentioned before, the patch allows you to override via configuration settings an impl defined as service. This will allow people to use alternate impls for a FS schema easily.

          Show
          Alejandro Abdelnur added a comment - @Owen, are you suggesting we should de-overlap schema names? If so, agreed. Some how related, it was mentioned before, the patch allows you to override via configuration settings an impl defined as service. This will allow people to use alternate impls for a FS schema easily.
          Hide
          Owen O'Malley added a comment -

          Sorry, I wasn't clear. Yes, part of using ServiceLoader to load these impls is that we need to make the schemes for each FileSystem unique. I was proposing that we change

          "file" scheme to point to RawLocalFileSystem
          "checksum" scheme to point to LocalFileSystem

          Show
          Owen O'Malley added a comment - Sorry, I wasn't clear. Yes, part of using ServiceLoader to load these impls is that we need to make the schemes for each FileSystem unique. I was proposing that we change "file" scheme to point to RawLocalFileSystem "checksum" scheme to point to LocalFileSystem
          Hide
          Suresh Srinivas added a comment -

          "file" scheme to point to RawLocalFileSystem, "checksum" scheme to point to LocalFileSystem

          This is backward incompatible. We should retain the definitions that are in core-default.xml for mapping scheme to a file system. By that definition, scheme "file" should map to LocalFileSystem. I am fine moving current schemes from configuration file to ServiceLoader based mechanism.

          We should also retain the configuration based mechanism. This as I indicated earlier, gives flexibility during run time to override a scheme to a different file system mapping.

          Show
          Suresh Srinivas added a comment - "file" scheme to point to RawLocalFileSystem, "checksum" scheme to point to LocalFileSystem This is backward incompatible. We should retain the definitions that are in core-default.xml for mapping scheme to a file system. By that definition, scheme "file" should map to LocalFileSystem. I am fine moving current schemes from configuration file to ServiceLoader based mechanism. We should also retain the configuration based mechanism. This as I indicated earlier, gives flexibility during run time to override a scheme to a different file system mapping.
          Hide
          Alejandro Abdelnur added a comment -

          Patch rebased to trunk.

          It keeps current mappings in the services definition (it was been done already).

          Definition in configuration overrides a service definition (it was been done already).

          It removes the deprecated warning log when resolving a FS implementation from configuration as this is the way of overriding an implementation, thus it is not deprecated.

          It adds HAR and WebHdfs filesystems to the service loading mechanism/definitions.

          Show
          Alejandro Abdelnur added a comment - Patch rebased to trunk. It keeps current mappings in the services definition (it was been done already). Definition in configuration overrides a service definition (it was been done already). It removes the deprecated warning log when resolving a FS implementation from configuration as this is the way of overriding an implementation, thus it is not deprecated. It adds HAR and WebHdfs filesystems to the service loading mechanism/definitions.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12514517/HADOOP-7549v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/596//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12514517/HADOOP-7549v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/596//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          test-patch fails with patches spanning across projects.

          Show
          Alejandro Abdelnur added a comment - test-patch fails with patches spanning across projects.
          Hide
          Suresh Srinivas added a comment -

          Alejandro, how does it handle the following:

          multiple implementations could be using the same scheme names (example RawLocalFileSystem, LocalFileSystem etc). I am not sure which one wins in that case. Also some one adding a jar to the path, with say an experimental file system (say HDFS) could override the DistributedFileSystem and not even know about it.

          Show
          Suresh Srinivas added a comment - Alejandro, how does it handle the following: multiple implementations could be using the same scheme names (example RawLocalFileSystem, LocalFileSystem etc). I am not sure which one wins in that case. Also some one adding a jar to the path, with say an experimental file system (say HDFS) could override the DistributedFileSystem and not even know about it.
          Hide
          Alejandro Abdelnur added a comment -

          @Suresh, If both impls using the same scheme are defined as services, then it is undetermined (by JDK specs). If one is defined as service an the other in the configuration, the one in the configuration wins. If both are defined in the configuration, the latter one wins.

          Show
          Alejandro Abdelnur added a comment - @Suresh, If both impls using the same scheme are defined as services, then it is undetermined (by JDK specs). If one is defined as service an the other in the configuration, the one in the configuration wins. If both are defined in the configuration, the latter one wins.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > test-patch fails with patches spanning across projects.

          This becomes a problem: patches spanning across projects usually are more involved but we run less tests on it. Is there a way to fix it?

          For the moment, we should manually check for the findbugs, javac and javadoc warnings, and run the unit tests before committing patches.

          Show
          Tsz Wo Nicholas Sze added a comment - > test-patch fails with patches spanning across projects. This becomes a problem: patches spanning across projects usually are more involved but we run less tests on it. Is there a way to fix it? For the moment, we should manually check for the findbugs, javac and javadoc warnings, and run the unit tests before committing patches.
          Hide
          Alejandro Abdelnur added a comment -

          rebasing to trunk to remove patch complains.

          I've run test-patch locally with and without the patch and I get the same number of warnings:

              +1 @author.  The patch does not contain any @author tags.
          
              +1 tests included.  The patch appears to include 2 new or modified test files.
          
              -1 javadoc.  The javadoc tool appears to have generated 8 warning messages.
          
              -1 javac.  The applied patch generated 1707 javac compiler warnings (more than the trunk's current 1706 warnings).
          
              +1 eclipse:eclipse.  The patch built with eclipse:eclipse.
          
              -1 findbugs.  The patch appears to introduce 20 new Findbugs (version 1.3.9) warnings.
          
              +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          
          Show
          Alejandro Abdelnur added a comment - rebasing to trunk to remove patch complains. I've run test-patch locally with and without the patch and I get the same number of warnings: +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. -1 javadoc. The javadoc tool appears to have generated 8 warning messages. -1 javac. The applied patch generated 1707 javac compiler warnings (more than the trunk's current 1706 warnings). +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 20 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings.
          Hide
          Alejandro Abdelnur added a comment -

          reattaching with correct patch filename

          Show
          Alejandro Abdelnur added a comment - reattaching with correct patch filename
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12523408/HADOOP-7549v3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/869//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523408/HADOOP-7549v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/869//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          test-patch failure due to cross module patch contents.

          Show
          Alejandro Abdelnur added a comment - test-patch failure due to cross module patch contents.
          Hide
          Tom White added a comment -

          +1 This looks good to me. Existing configurations will still work with this patch, so there are no compatibility concerns.

          Show
          Tom White added a comment - +1 This looks good to me. Existing configurations will still work with this patch, so there are no compatibility concerns.
          Hide
          Alejandro Abdelnur added a comment -

          committed to trunk and branch-2. (committed v2 as v3 which is identical was missing the service files)

          Show
          Alejandro Abdelnur added a comment - committed to trunk and branch-2. (committed v2 as v3 which is identical was missing the service files)
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2200 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2200/)
          HADOOP-7549. Use JDK ServiceLoader mechanism to find FileSystem implementations. (tucu) (Revision 1329994)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329994
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/kfs/KosmosFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCaching.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2200 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2200/ ) HADOOP-7549 . Use JDK ServiceLoader mechanism to find FileSystem implementations. (tucu) (Revision 1329994) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329994 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/kfs/KosmosFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCaching.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2126 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2126/)
          HADOOP-7549. Use JDK ServiceLoader mechanism to find FileSystem implementations. (tucu) (Revision 1329994)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329994
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/kfs/KosmosFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCaching.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2126 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2126/ ) HADOOP-7549 . Use JDK ServiceLoader mechanism to find FileSystem implementations. (tucu) (Revision 1329994) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329994 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/kfs/KosmosFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCaching.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2142 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2142/)
          HADOOP-7549. Use JDK ServiceLoader mechanism to find FileSystem implementations. (tucu) (Revision 1329994)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329994
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/kfs/KosmosFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCaching.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2142 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2142/ ) HADOOP-7549 . Use JDK ServiceLoader mechanism to find FileSystem implementations. (tucu) (Revision 1329994) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329994 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/kfs/KosmosFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCaching.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1025 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1025/)
          HADOOP-7549. Use JDK ServiceLoader mechanism to find FileSystem implementations. (tucu) (Revision 1329994)

          Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329994
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/kfs/KosmosFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCaching.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1025 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1025/ ) HADOOP-7549 . Use JDK ServiceLoader mechanism to find FileSystem implementations. (tucu) (Revision 1329994) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329994 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/kfs/KosmosFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCaching.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1060 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1060/)
          HADOOP-7549. Use JDK ServiceLoader mechanism to find FileSystem implementations. (tucu) (Revision 1329994)

          Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329994
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/kfs/KosmosFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCaching.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1060 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1060/ ) HADOOP-7549 . Use JDK ServiceLoader mechanism to find FileSystem implementations. (tucu) (Revision 1329994) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329994 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/kfs/KosmosFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCaching.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
          Hide
          Tsz Wo Nicholas Sze added a comment -

          It seems that this breaks FsUrlStreamHandlerFactory; see also HDFS-3323.

          Show
          Tsz Wo Nicholas Sze added a comment - It seems that this breaks FsUrlStreamHandlerFactory; see also HDFS-3323 .
          Hide
          Jane Chen added a comment -

          Hi Alejandro,

          I was calling FileSystem.get(URI, conf, user) from C++ code through JNI. After upgrading to 2.0, this call is giving me "java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.fs.LocalFileSystem not found" error. It doesn't seem to be a classpath issue, since I can instantiate the class with no problem. I'd like to know whether you have any insight on this.

          Also I read in the bug that there's a configuration to disable loading through ServiceLoader. I'd like to try that, and would appreciate if you could let me know what's the proper way to do that.

          Thanks,
          Jane

          Show
          Jane Chen added a comment - Hi Alejandro, I was calling FileSystem.get(URI, conf, user) from C++ code through JNI. After upgrading to 2.0, this call is giving me "java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.fs.LocalFileSystem not found" error. It doesn't seem to be a classpath issue, since I can instantiate the class with no problem. I'd like to know whether you have any insight on this. Also I read in the bug that there's a configuration to disable loading through ServiceLoader. I'd like to try that, and would appreciate if you could let me know what's the proper way to do that. Thanks, Jane
          Hide
          Alejandro Abdelnur added a comment -

          Jane,

          If you are unjaring/rejaring Hadoop JARs when using Hadoop from a JVM started from C++ then that could be the issue. JDK ServiceLoader uses the same file name in multiple JARs to pick up all services. If you unjar/rejar the JARs, you are looking all but one of those files.

          You could try using the old mechanism for registering filesystems in the hadoop configuration files 'fs.<SCHEME>.impl=<CLASS>.

          Show
          Alejandro Abdelnur added a comment - Jane, If you are unjaring/rejaring Hadoop JARs when using Hadoop from a JVM started from C++ then that could be the issue. JDK ServiceLoader uses the same file name in multiple JARs to pick up all services. If you unjar/rejar the JARs, you are looking all but one of those files. You could try using the old mechanism for registering filesystems in the hadoop configuration files 'fs.<SCHEME>.impl=<CLASS>.
          Hide
          Jane Chen added a comment -

          Hi Alejandro,

          Thank you for your input.

          I was using the bundled jars directly. I found that I have to call FileSystem.loadFileSystems() in the same thread where JVM is initialized. So I got it working now.

          I'm guessing the class loader used in the ServiceLoader doesn't get the classpath set if it isn't the same thread creating the JVM. But I haven't tried to prove it. I'd love to hear it if you have any insight on that.

          Thanks again,
          Jane

          Show
          Jane Chen added a comment - Hi Alejandro, Thank you for your input. I was using the bundled jars directly. I found that I have to call FileSystem.loadFileSystems() in the same thread where JVM is initialized. So I got it working now. I'm guessing the class loader used in the ServiceLoader doesn't get the classpath set if it isn't the same thread creating the JVM. But I haven't tried to prove it. I'd love to hear it if you have any insight on that. Thanks again, Jane

            People

            • Assignee:
              Alejandro Abdelnur
              Reporter:
              Alejandro Abdelnur
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development