HBase
  1. HBase
  2. HBASE-10248

[0.94] TestMetaReaderEditor fails frequently

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Eample:

      java.lang.AssertionError: reader: count=2, t=null
      	at org.junit.Assert.fail(Assert.java:93)
      	at org.junit.Assert.assertTrue(Assert.java:43)
      	at org.apache.hadoop.hbase.catalog.TestMetaReaderEditor.testRetrying(TestMetaReaderEditor.java:152)
      

      https://builds.apache.org/job/HBase-0.94/1239/

        Activity

        Hide
        Lars Hofhansl added a comment -

        I think the key is this from the log:

        2013-12-26 07:56:22,201 INFO  [reader] catalog.TestMetaReaderEditor$MetaTask(192): reader failed
        org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=10, exceptions:
        Thu Dec 26 07:55:11 UTC 2013, org.apache.hadoop.hbase.client.HTable$3@97cc12, java.io.IOException: Unexpected closed connection
        Thu Dec 26 07:55:12 UTC 2013, org.apache.hadoop.hbase.client.HTable$3@97cc12, org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: janus.apache.org/67.195.138.60:50758
        Thu Dec 26 07:55:13 UTC 2013, org.apache.hadoop.hbase.client.HTable$3@97cc12, org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: janus.apache.org/67.195.138.60:50758
        Thu Dec 26 07:55:15 UTC 2013, org.apache.hadoop.hbase.client.HTable$3@97cc12, org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: janus.apache.org/67.195.138.60:50758
        Thu Dec 26 07:55:17 UTC 2013, org.apache.hadoop.hbase.client.HTable$3@97cc12, org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: janus.apache.org/67.195.138.60:50758
        
        Show
        Lars Hofhansl added a comment - I think the key is this from the log: 2013-12-26 07:56:22,201 INFO [reader] catalog.TestMetaReaderEditor$MetaTask(192): reader failed org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=10, exceptions: Thu Dec 26 07:55:11 UTC 2013, org.apache.hadoop.hbase.client.HTable$3@97cc12, java.io.IOException: Unexpected closed connection Thu Dec 26 07:55:12 UTC 2013, org.apache.hadoop.hbase.client.HTable$3@97cc12, org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: janus.apache.org/67.195.138.60:50758 Thu Dec 26 07:55:13 UTC 2013, org.apache.hadoop.hbase.client.HTable$3@97cc12, org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: janus.apache.org/67.195.138.60:50758 Thu Dec 26 07:55:15 UTC 2013, org.apache.hadoop.hbase.client.HTable$3@97cc12, org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: janus.apache.org/67.195.138.60:50758 Thu Dec 26 07:55:17 UTC 2013, org.apache.hadoop.hbase.client.HTable$3@97cc12, org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: janus.apache.org/67.195.138.60:50758
        Hide
        Lars Hofhansl added a comment - - edited
        client.HConnectionManager$HConnectionImplementation(1738): Failed all from region=.META.,,1.1028785192, hostname=janus.apache.org, port=50758
        java.util.concurrent.ExecutionException: java.io.IOException: Unexpected closed connection
        	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
        	at java.util.concurrent.FutureTask.get(FutureTask.java:83)
        	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1708)
        	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1560)
        	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:994)
        	at org.apache.hadoop.hbase.client.HTable.put(HTable.java:828)
        	at org.apache.hadoop.hbase.catalog.MetaEditor.put(MetaEditor.java:102)
        	at org.apache.hadoop.hbase.catalog.MetaEditor.putToCatalogTable(MetaEditor.java:92)
        	at org.apache.hadoop.hbase.catalog.MetaEditor.updateLocation(MetaEditor.java:296)
        	at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:276)
        	at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1874)
        	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:240)
        Caused by: java.io.IOException: Unexpected closed connection
        	at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
        	at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:87)
        	at com.sun.proxy.$Proxy21.multi(Unknown Source)
        	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1537)
        	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1535)
        	at org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:229)
        	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1544)
        	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1532)
        
        Show
        Lars Hofhansl added a comment - - edited client.HConnectionManager$HConnectionImplementation(1738): Failed all from region=.META.,,1.1028785192, hostname=janus.apache.org, port=50758 java.util.concurrent.ExecutionException: java.io.IOException: Unexpected closed connection at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1708) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1560) at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:994) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:828) at org.apache.hadoop.hbase.catalog.MetaEditor.put(MetaEditor.java:102) at org.apache.hadoop.hbase.catalog.MetaEditor.putToCatalogTable(MetaEditor.java:92) at org.apache.hadoop.hbase.catalog.MetaEditor.updateLocation(MetaEditor.java:296) at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:276) at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1874) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:240) Caused by: java.io.IOException: Unexpected closed connection at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:87) at com.sun.proxy.$Proxy21.multi(Unknown Source) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1537) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1535) at org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:229) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1544) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1532)
        Hide
        Lars Hofhansl added a comment -

        And:

        2013-12-26 07:55:12,739 WARN  [PostOpenDeployTasks:a901060a2b255c18e70abb771ae6c9fd] client.HConnectionManager$HConnectionImplementation(1738): Failed all from region=.META.,,1.1028785192, hostname=janus.apache.org, port=50758
        java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: janus.apache.org/67.195.138.60:50758
        	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
        	at java.util.concurrent.FutureTask.get(FutureTask.java:83)
        	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1708)
        	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1560)
        	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:994)
        	at org.apache.hadoop.hbase.client.HTable.put(HTable.java:828)
        	at org.apache.hadoop.hbase.catalog.MetaEditor.put(MetaEditor.java:102)
        	at org.apache.hadoop.hbase.catalog.MetaEditor.putToCatalogTable(MetaEditor.java:92)
        	at org.apache.hadoop.hbase.catalog.MetaEditor.updateLocation(MetaEditor.java:296)
        	at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:276)
        	at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1874)
        	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:240)
        Caused by: org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: janus.apache.org/67.195.138.60:50758
        	at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:427)
        	at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1141)
        	at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:988)
        	at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:87)
        	at com.sun.proxy.$Proxy21.multi(Unknown Source)
        	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1537)
        	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1535)
        	at org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:229)
        	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1544)
        	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1532)
        
        Show
        Lars Hofhansl added a comment - And: 2013-12-26 07:55:12,739 WARN [PostOpenDeployTasks:a901060a2b255c18e70abb771ae6c9fd] client.HConnectionManager$HConnectionImplementation(1738): Failed all from region=.META.,,1.1028785192, hostname=janus.apache.org, port=50758 java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: janus.apache.org/67.195.138.60:50758 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1708) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1560) at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:994) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:828) at org.apache.hadoop.hbase.catalog.MetaEditor.put(MetaEditor.java:102) at org.apache.hadoop.hbase.catalog.MetaEditor.putToCatalogTable(MetaEditor.java:92) at org.apache.hadoop.hbase.catalog.MetaEditor.updateLocation(MetaEditor.java:296) at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:276) at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1874) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:240) Caused by: org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: janus.apache.org/67.195.138.60:50758 at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:427) at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1141) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:988) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:87) at com.sun.proxy.$Proxy21.multi(Unknown Source) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1537) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1535) at org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:229) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1544) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1532)
        Hide
        Lars Hofhansl added a comment -

        This looks like an actual problem (not just a test issue), where an HBaseClient is rendered permanently useless.

        Show
        Lars Hofhansl added a comment - This looks like an actual problem (not just a test issue), where an HBaseClient is rendered permanently useless.
        Hide
        Lars Hofhansl added a comment -

        Actually this is the reason:

        2013-12-26 07:55:11,646 FATAL [MASTER_CLOSE_REGION-janus.apache.org,36597,1388044485155-1] master.HMaster(1621): Master server abort: loaded coprocessors are: []
        2013-12-26 07:55:11,646 FATAL [MASTER_CLOSE_REGION-janus.apache.org,36597,1388044485155-1] master.HMaster(1626): Unexpected state : testRetrying,ttt,1388044498231.50d8f699ee870d7af05aa4f4b6824e8c. state=PENDING_OPEN, ts=1388044511646, server=janus.apache.org,42769,1388044485732 .. Cannot transit it to OFFLINE.
        java.lang.IllegalStateException: Unexpected state : testRetrying,ttt,1388044498231.50d8f699ee870d7af05aa4f4b6824e8c. state=PENDING_OPEN, ts=1388044511646, server=janus.apache.org,42769,1388044485732 .. Cannot transit it to OFFLINE.
        	at org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1890)
        	at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1690)
        	at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1426)
        	at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1398)
        	at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1393)
        	at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
        	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        	at java.lang.Thread.run(Thread.java:662)
        
        Show
        Lars Hofhansl added a comment - Actually this is the reason: 2013-12-26 07:55:11,646 FATAL [MASTER_CLOSE_REGION-janus.apache.org,36597,1388044485155-1] master.HMaster(1621): Master server abort: loaded coprocessors are: [] 2013-12-26 07:55:11,646 FATAL [MASTER_CLOSE_REGION-janus.apache.org,36597,1388044485155-1] master.HMaster(1626): Unexpected state : testRetrying,ttt,1388044498231.50d8f699ee870d7af05aa4f4b6824e8c. state=PENDING_OPEN, ts=1388044511646, server=janus.apache.org,42769,1388044485732 .. Cannot transit it to OFFLINE. java.lang.IllegalStateException: Unexpected state : testRetrying,ttt,1388044498231.50d8f699ee870d7af05aa4f4b6824e8c. state=PENDING_OPEN, ts=1388044511646, server=janus.apache.org,42769,1388044485732 .. Cannot transit it to OFFLINE. at org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1890) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1690) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1426) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1398) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1393) at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang. Thread .run( Thread .java:662)
        Hide
        Lars Hofhansl added a comment -

        So this is related to HBASE-8912.

        Show
        Lars Hofhansl added a comment - So this is related to HBASE-8912 .

          People

          • Assignee:
            Unassigned
            Reporter:
            Lars Hofhansl
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development