HBase
  1. HBase
  2. HBASE-170

Web UI displays an IOException instead of the Tables

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      For me after every second restart I get an error when loading the Hbase UI. Here the page:

      Master: 192.168.105.11:60000
      HQL, Local logs, Thread Dump, Log Level
      ______________________________________________________________________________________________________________________

      Master Attributes
      Attribute Name Value Description
      Filesystem lv1-xen-pdc-2.worldlingo.com:9000 Filesystem hbase is running on
      Hbase Root Directory /hbase Location of hbase home directory

      Online META Regions

      Name Server
      ROOT 192.168.105.31:60020
      .META.,,1 192.168.105.39:60020

      Tables

      error msg : java.io.IOException: java.io.IOException: HStoreScanner failed construction at
      org.apache.hadoop.hbase.HStore$StoreFileScanner.(HStore.java:1879) at
      org.apache.hadoop.hbase.HStore$HStoreScanner.(HStore.java:2000) at
      org.apache.hadoop.hbase.HStore.getScanner(HStore.java:1822) at
      org.apache.hadoop.hbase.HRegion$HScanner.(HRegion.java:1543) at
      org.apache.hadoop.hbase.HRegion.getScanner(HRegion.java:1118) at
      org.apache.hadoop.hbase.HRegionServer.openScanner(HRegionServer.java:1465) at
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at
      java.lang.reflect.Method.invoke(Method.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:401) at
      org.apache.hadoop.ipc.Server$Handler.run(Server.java:892) Caused by: org.apache.hadoop.ipc.RemoteException:
      java.io.IOException: File does not exist: /hbase/hregion_1028785192/info/mapfiles/6628785818889695133/data at
      org.apache.hadoop.dfs.FSDirectory.getFileInfo(FSDirectory.java:489) at
      org.apache.hadoop.dfs.FSNamesystem.getFileInfo(FSNamesystem.java:1380) at
      org.apache.hadoop.dfs.NameNode.getFileInfo(NameNode.java:425) at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown
      Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at
      java.lang.reflect.Method.invoke(Method.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:401) at
      org.apache.hadoop.ipc.Server$Handler.run(Server.java:892) at org.apache.hadoop.ipc.Client.call(Client.java:509) at
      org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:198) at org.apache.hadoop.dfs.$Proxy1.getFileInfo(Unknown Source) at
      sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at
      java.lang.reflect.Method.invoke(Method.java:585) at
      org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) at
      org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) at
      org.apache.hadoop.dfs.$Proxy1.getFileInfo(Unknown Source) at
      org.apache.hadoop.dfs.DFSClient.getFileInfo(DFSClient.java:416) at
      org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:331) at
      org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:492) at
      org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1349) at
      org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1344) at org.apache.hadoop.io.MapFile$Reader.(MapFile.java:254) at org.apache.hadoop.io.MapFile$Reader.(MapFile.java:242) at
      org.apache.hadoop.hbase.HStoreFile$BloomFilterMapFile$Reader.(HStoreFile.java:816) at org.apache.hadoop.hbase.HStoreFile.getReader(HStoreFile.java:936) at
      org.apache.hadoop.hbase.HStore$StoreFileScanner.(HStore.java:1854) ... 11 more

      Region Servers
      Address Start Code Load
      192.168.105.41:60020 1200095856575 requests: 0 regions: 48
      192.168.105.31:60020 1200095853516 requests: 0 regions: 38
      192.168.105.51:60020 1200095856709 requests: 0 regions: 20
      192.168.105.21:60020 1200095852330 requests: 0 regions: 22
      192.168.105.61:60020 1200095856758 requests: 0 regions: 39
      192.168.105.71:60020 1200095856975 requests: 0 regions: 21
      192.168.105.53:60020 1200095856655 requests: 0 regions: 19
      192.168.105.43:60020 1200095854323 requests: 0 regions: 25
      192.168.105.33:60020 1200095853508 requests: 0 regions: 218
      192.168.105.23:60020 1200095852282 requests: 0 regions: 26
      192.168.105.13:60020 1200095851830 requests: 0 regions: 27
      192.168.105.73:60020 1200095856843 requests: 0 regions: 21
      192.168.105.63:60020 1200095856776 requests: 0 regions: 19
      192.168.105.35:60020 1200095853520 requests: 0 regions: 24
      192.168.105.25:60020 1200095853591 requests: 1 regions: 25
      192.168.105.75:60020 1200095856884 requests: 0 regions: 37
      192.168.105.65:60020 1200095856817 requests: 0 regions: 19
      192.168.105.55:60020 1200095856625 requests: 0 regions: 31
      192.168.105.27:60020 1200095853490 requests: 0 regions: 23 192.168.105.37:60020 1200095853876 requests: 0 regions: 21
      192.168.105.17:60020 1200095851832 requests: 0 regions: 25
      192.168.105.47:60020 1200095856542 requests: 0 regions: 20
      192.168.105.77:60020 1200095856853 requests: 0 regions: 22
      192.168.105.67:60020 1200095856853 requests: 0 regions: 19
      192.168.105.57:60020 1200095856719 requests: 0 regions: 25
      192.168.105.19:60020 1200095851818 requests: 0 regions: 29
      192.168.105.39:60020 1200095853864 requests: 0 regions: 2
      192.168.105.29:60020 1200095853521 requests: 0 regions: 25
      192.168.105.69:60020 1200095856831 requests: 0 regions: 21
      192.168.105.49:60020 1200095856553 requests: 0 regions: 36
      192.168.105.59:60020 1200095856736 requests: 0 regions: 19
      Total: servers: 33 requests: 1 regions: 1017

      Load is requests per hbase.regionsserver.msginterval (3 second(s)) and count of regions loaded

      After another restart, the tables display as they should. Reloading the page does not help.

        Activity

        Hide
        stack added a comment -

        Any other context in the master logs that you might think of use Lars? (Are you running w/ DEBUG enabled? If not, see hbase FAQ for how).

        Show
        stack added a comment - Any other context in the master logs that you might think of use Lars? (Are you running w/ DEBUG enabled? If not, see hbase FAQ for how).
        Hide
        Bryan Duxbury added a comment -

        This issue appears to be a cosmetic ui interface from what I can tell. I'm reducing the priority to match this problem.

        Show
        Bryan Duxbury added a comment - This issue appears to be a cosmetic ui interface from what I can tell. I'm reducing the priority to match this problem.
        Hide
        stack added a comment -

        Fix is to have listTables retry. Cache had old region value and was never getting cleared.

        M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HConnectionManager.java
        Add retry getting list of tables.
        M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HBaseAdmin.java
        Remove unused imports.
        M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HTable.java
        Removed unused imports.
        (relocateRegion): Javadoc for reload param and actually use it.
        M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMaster.java
        (getConfiguration): Added.
        M src/contrib/hbase/src/java/org/apache/hadoop/hbase/generated/regionserver/regionserver_jsp.java
        M src/contrib/hbase/src/java/org/apache/hadoop/hbase/generated/master/master_jsp.java
        Regenerated.
        M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegion.java
        Renamed local variables that had same names as data members to avoid
        confusion. Get 'clone' of Writable from hadoop WritableUtils rather than
        from the copy that used to be Writables.
        M src/contrib/hbase/src/java/org/apache/hadoop/hbase/util/Writables.java
        Moved the clone method copied from WritableUtils (HADOOP-2650)
        M src/contrib/hbase/src/webapps/regionserver/regionserver.jsp
        Add in version and build info here too (Could be different from the master)
        M src/contrib/hbase/src/webapps/master/hql.jsp
        Fix import.
        M src/contrib/hbase/src/webapps/master/master.jsp
        Get the conf from master rather than create one on each reload.

        Show
        stack added a comment - Fix is to have listTables retry. Cache had old region value and was never getting cleared. M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HConnectionManager.java Add retry getting list of tables. M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HBaseAdmin.java Remove unused imports. M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HTable.java Removed unused imports. (relocateRegion): Javadoc for reload param and actually use it. M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMaster.java (getConfiguration): Added. M src/contrib/hbase/src/java/org/apache/hadoop/hbase/generated/regionserver/regionserver_jsp.java M src/contrib/hbase/src/java/org/apache/hadoop/hbase/generated/master/master_jsp.java Regenerated. M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegion.java Renamed local variables that had same names as data members to avoid confusion. Get 'clone' of Writable from hadoop WritableUtils rather than from the copy that used to be Writables. M src/contrib/hbase/src/java/org/apache/hadoop/hbase/util/Writables.java Moved the clone method copied from WritableUtils ( HADOOP-2650 ) M src/contrib/hbase/src/webapps/regionserver/regionserver.jsp Add in version and build info here too (Could be different from the master) M src/contrib/hbase/src/webapps/master/hql.jsp Fix import. M src/contrib/hbase/src/webapps/master/master.jsp Get the conf from master rather than create one on each reload.
        Hide
        stack added a comment -

        Passes tests locally.

        Show
        stack added a comment - Passes tests locally.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12373615/ui.patch
        against trunk revision r613499.

        @author +1. The patch does not contain any @author tags.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new compiler warnings.

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests +1. The patch passed core unit tests.

        contrib tests +1. The patch passed contrib unit tests.

        Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1666/testReport/
        Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1666/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1666/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1666/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12373615/ui.patch against trunk revision r613499. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1666/testReport/ Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1666/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1666/artifact/trunk/build/test/checkstyle-errors.html Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1666/console This message is automatically generated.
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-Nightly #373 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/373/ )
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-Nightly #374 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/374/ )
        Hide
        stack added a comment -

        Committed (a while back). Resolving.

        Show
        stack added a comment - Committed (a while back). Resolving.

          People

          • Assignee:
            Unassigned
            Reporter:
            Lars George
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development