HBase
  1. HBase
  2. HBASE-5964

HFileSystem: "No FileSystem for scheme: hdfs"

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.94.0, 0.95.2
    • Fix Version/s: 0.94.1, 0.95.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      Hadoop 2.0.0-alpha-SNAPSHOT, HBase 0.94.0-SNAPSHOT

    • Hadoop Flags:
      Reviewed

      Description

      I've been seeing this with Hadoop 2.0.0-alpha-SNAPSHOT and HBase 0.94.0-SNAPSHOT:

      2012-05-08 15:18:00,692 FATAL [RegionServer:0;acer.localdomain,48307,1336515479011] regionserver.HRegionServer(1674): ABORTING region server acer.localdomain,48307,1336515479011: Unhandled exception: No FileSystem for scheme: hdfs
      java.io.IOException: No FileSystem for scheme: hdfs
      	at org.apache.hadoop.hbase.fs.HFileSystem.newInstanceFileSystem(HFileSystem.java:146)
      	at org.apache.hadoop.hbase.fs.HFileSystem.<init>(HFileSystem.java:75)
      	at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:973)
      	at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.handleReportForDutyResponse(MiniHBaseCluster.java:110)
      	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:671)
      	at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:136)
      	at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:89)
      	at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:120)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:357)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:616)
      	at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
      	at org.apache.hadoop.hbase.security.User.call(User.java:586)
      	at org.apache.hadoop.hbase.security.User.access$700(User.java:50)
      	at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:426)
      	at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:118)
      	at java.lang.Thread.run(Thread.java:679)
      

      Not sure precisely when it started. First I thought it might be due to HBASE-5955 but I cherry picked that change over from trunk. Then I got HBASE-5963 out of the way.

      1. HBASE-5964.patch
        2 kB
        Andrew Purtell

        Activity

        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security #27 (See https://builds.apache.org/job/HBase-0.94-security/27/)
        HBASE-5964. HFileSystem: "No FileSystem for scheme: hdfs" (Revision 1336345)

        Result = SUCCESS
        apurtell :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security #27 (See https://builds.apache.org/job/HBase-0.94-security/27/ ) HBASE-5964 . HFileSystem: "No FileSystem for scheme: hdfs" (Revision 1336345) Result = SUCCESS apurtell : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-security #197 (See https://builds.apache.org/job/HBase-TRUNK-security/197/)
        HBASE-5964. HFileSystem: "No FileSystem for scheme: hdfs" (Revision 1336343)

        Result = FAILURE
        apurtell :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-security #197 (See https://builds.apache.org/job/HBase-TRUNK-security/197/ ) HBASE-5964 . HFileSystem: "No FileSystem for scheme: hdfs" (Revision 1336343) Result = FAILURE apurtell : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94 #185 (See https://builds.apache.org/job/HBase-0.94/185/)
        HBASE-5964. HFileSystem: "No FileSystem for scheme: hdfs" (Revision 1336345)

        Result = FAILURE
        apurtell :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java
        Show
        Hudson added a comment - Integrated in HBase-0.94 #185 (See https://builds.apache.org/job/HBase-0.94/185/ ) HBASE-5964 . HFileSystem: "No FileSystem for scheme: hdfs" (Revision 1336345) Result = FAILURE apurtell : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2859 (See https://builds.apache.org/job/HBase-TRUNK/2859/)
        HBASE-5964. HFileSystem: "No FileSystem for scheme: hdfs" (Revision 1336343)

        Result = FAILURE
        apurtell :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2859 (See https://builds.apache.org/job/HBase-TRUNK/2859/ ) HBASE-5964 . HFileSystem: "No FileSystem for scheme: hdfs" (Revision 1336343) Result = FAILURE apurtell : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java
        Hide
        Andrew Purtell added a comment -

        Committed to TRUNK and 0.94 branch. Thanks for the review Ted.

        Show
        Andrew Purtell added a comment - Committed to TRUNK and 0.94 branch. Thanks for the review Ted.
        Hide
        Ted Yu added a comment -

        +1 on commit.

        Show
        Ted Yu added a comment - +1 on commit.
        Hide
        Andrew Purtell added a comment -

        We should commit this, doing anything else with recent branch-2 requires it.

        Show
        Andrew Purtell added a comment - We should commit this, doing anything else with recent branch-2 requires it.
        Hide
        ramkrishna.s.vasudevan added a comment -

        Thanks for your reply.
        So let me cross check once again then why there was a leak in HDFS DN. I heard there were some fixes done on that side in HDFS related to connection caching.

        Show
        ramkrishna.s.vasudevan added a comment - Thanks for your reply. So let me cross check once again then why there was a leak in HDFS DN. I heard there were some fixes done on that side in HDFS related to connection caching.
        Hide
        Andrew Purtell added a comment -

        That configuration parameter disables FS instance caching. This is required to preserve the semantics of the HBase utility method.

        Show
        Andrew Purtell added a comment - That configuration parameter disables FS instance caching. This is required to preserve the semantics of the HBase utility method.
        Hide
        Ted Yu added a comment -

        The default is true in hadoop. A few examples:

            conf.set("fs.hdfs.impl.disable.cache", "true");
        ./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/service/hadoop/FileSystemAccessService.java
                    clientConf.setBoolean("fs.hdfs.impl.disable.cache", true);
        ./hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/DistributedRaidFileSystem.java
        
        Show
        Ted Yu added a comment - The default is true in hadoop. A few examples: conf.set( "fs.hdfs.impl.disable.cache" , " true " ); ./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/service/hadoop/FileSystemAccessService.java clientConf.setBoolean( "fs.hdfs.impl.disable.cache" , true ); ./hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/DistributedRaidFileSystem.java
        Hide
        ramkrishna.s.vasudevan added a comment -

        In our testing of 0.94RC with a bit older version of hadoop -2.0.0 we got filehandler leak in DN leading to OOME. Later we found that the same has been fixed very recently in HDFS.

        AFter taking the latest hadoopp-2.0.0-alpha release we got all the issues that Andy raised. Now filehandler leak does not happen when the patch of this JIRA is included

        clone.setBoolean("fs." + uri.getScheme() + ".impl.disable.cache", true);
        

        But why this property is made by default to 'true'? Is it by default true in Hadoop also?

        Show
        ramkrishna.s.vasudevan added a comment - In our testing of 0.94RC with a bit older version of hadoop -2.0.0 we got filehandler leak in DN leading to OOME. Later we found that the same has been fixed very recently in HDFS. AFter taking the latest hadoopp-2.0.0-alpha release we got all the issues that Andy raised. Now filehandler leak does not happen when the patch of this JIRA is included clone.setBoolean( "fs." + uri.getScheme() + ".impl.disable.cache" , true ); But why this property is made by default to 'true'? Is it by default true in Hadoop also?
        Hide
        Ted Yu added a comment -

        Probably it was specific to the build machine.
        Resubmitting the patch would give us some idea.

        Show
        Ted Yu added a comment - Probably it was specific to the build machine. Resubmitting the patch would give us some idea.
        Hide
        Andrew Purtell added a comment -

        You think that somehow related to this patch Ted?

        Show
        Andrew Purtell added a comment - You think that somehow related to this patch Ted?
        Hide
        Ted Yu added a comment -

        My script wasn't able to identify the hanging test.
        I saw the following from https://builds.apache.org/job/PreCommit-HBASE-Build/1805/console:

        Running org.apache.hadoop.hbase.io.hfile.TestLruBlockCache
        Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.823 sec
        #
        # There is insufficient memory for the Java Runtime Environment to continue.
        # Native memory allocation (malloc) failed to allocate 32756 bytes for ChunkPool::allocate
        # An error report file with more information is saved as:
        # /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hs_err_pid19988.log
        Aborted
        org.apache.maven.surefire.booter.SurefireBooterForkException: Error occurred in starting fork, check output in log
        	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:238)
        	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$000(ForkStarter.java:64)
        	at org.apache.maven.plugin.surefire.booterclient.ForkStarter$ParallelFork.call(ForkStarter.java:303)
        	at org.apache.maven.plugin.surefire.booterclient.ForkStarter$ParallelFork.call(ForkStarter.java:285)
        	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        	at java.lang.Thread.run(Thread.java:662)
        Running org.apache.hadoop.hbase.io.hfile.TestHFileSeek
        
        Show
        Ted Yu added a comment - My script wasn't able to identify the hanging test. I saw the following from https://builds.apache.org/job/PreCommit-HBASE-Build/1805/console: Running org.apache.hadoop.hbase.io.hfile.TestLruBlockCache Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.823 sec # # There is insufficient memory for the Java Runtime Environment to continue . # Native memory allocation (malloc) failed to allocate 32756 bytes for ChunkPool::allocate # An error report file with more information is saved as: # /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hs_err_pid19988.log Aborted org.apache.maven.surefire.booter.SurefireBooterForkException: Error occurred in starting fork, check output in log at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:238) at org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$000(ForkStarter.java:64) at org.apache.maven.plugin.surefire.booterclient.ForkStarter$ParallelFork.call(ForkStarter.java:303) at org.apache.maven.plugin.surefire.booterclient.ForkStarter$ParallelFork.call(ForkStarter.java:285) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang. Thread .run( Thread .java:662) Running org.apache.hadoop.hbase.io.hfile.TestHFileSeek
        Hide
        Andrew Purtell added a comment -

        The test report for build 1805 shows no test failures, so WTF. For the record I'm ignoring Hadoop QA.

        Show
        Andrew Purtell added a comment - The test report for build 1805 shows no test failures, so WTF. For the record I'm ignoring Hadoop QA.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12526067/HBASE-5964.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1805//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1805//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1805//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526067/HBASE-5964.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1805//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1805//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1805//console This message is automatically generated.

          People

          • Assignee:
            Andrew Purtell
            Reporter:
            Andrew Purtell
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development