Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-1148

Somewhat misleading error message when SYSTEM.CATALOG is disabled

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 4.0.0
    • None
    • None
    • None

    Description

      I was helping someone today whose HBase environment was a bit messed up. Somehow the system tables had gotten disabled. The error message threw us off enough that I think it should be improved. The main problem is that this message appears in red text, calling the most attention to itself:

      Error: ERROR 2006 (INT08): Incompatible jars detected between client and server. Ensure that phoenix.jar is put on the classpath of HBase in every region server: SYSTEM.CATALOG is disabled. (state=INT08,code=2006)

      Based on this message we started checking for duplicate JARs, etc. Now admittedly the real problem is stated there along with the misleading error message but it was enough to throw us off for a while.

      Reproduction is easy: disable your SYSTEM.CATALOG and connect using sqlline.

      Full traceback is attached, for the curious:

      org.apache.hadoop.hbase.TableNotEnabledException: SYSTEM.CATALOG is disabled.
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1094)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1084)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:904)
	at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:78)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:120)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:97)
	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:90)
	at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:51)
	at org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService$Stub.getVersion(MetaDataProtos.java:6805)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl$3.call(ConnectionQueryServicesImpl.java:909)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl$3.call(ConnectionQueryServicesImpl.java:901)
	at org.apache.hadoop.hbase.client.HTable$16.call(HTable.java:1586)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/07/15 07:41:37 WARN client.HTable: Error calling coprocessor service org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService for row
java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.TableNotEnabledException: SYSTEM.CATALOG is disabled.
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:188)
	at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1598)
	at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1554)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl.checkClientServerCompatibility(ConnectionQueryServicesImpl.java:900)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:825)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:1058)
	at org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:1156)
	at org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:422)
	at org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:183)
	at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:226)
	at org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:908)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:1453)
	at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:131)
	at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.connect(PhoenixEmbeddedDriver.java:112)
	at sqlline.SqlLine$DatabaseConnection.connect(SqlLine.java:4650)
	at sqlline.SqlLine$DatabaseConnection.getConnection(SqlLine.java:4701)
	at sqlline.SqlLine$Commands.connect(SqlLine.java:3942)
	at sqlline.SqlLine$Commands.connect(SqlLine.java:3851)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at sqlline.SqlLine$ReflectiveCommandHandler.execute(SqlLine.java:2810)
	at sqlline.SqlLine.dispatch(SqlLine.java:817)
	at sqlline.SqlLine.initArgs(SqlLine.java:633)
	at sqlline.SqlLine.begin(SqlLine.java:680)
	at sqlline.SqlLine.mainWithInputRedirection(SqlLine.java:441)
	at sqlline.SqlLine.main(SqlLine.java:424)
Caused by: org.apache.hadoop.hbase.TableNotEnabledException: SYSTEM.CATALOG is disabled.
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1094)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1084)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:904)
	at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:78)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:120)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:97)
	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:90)
	at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:51)
	at org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService$Stub.getVersion(MetaDataProtos.java:6805)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl$3.call(ConnectionQueryServicesImpl.java:909)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl$3.call(ConnectionQueryServicesImpl.java:901)
	at org.apache.hadoop.hbase.client.HTable$16.call(HTable.java:1586)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
Error: ERROR 2006 (INT08): Incompatible jars detected between client and server. Ensure that phoenix.jar is put on the classpath of HBase in every region server: SYSTEM.CATALOG is disabled. (state=INT08,code=2006)
java.sql.SQLException: ERROR 2006 (INT08): Incompatible jars detected between client and server. Ensure that phoenix.jar is put on the classpath of HBase in every region server: SYSTEM.CATALOG is disabled.
	at org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:309)
	at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:133)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl.checkClientServerCompatibility(ConnectionQueryServicesImpl.java:932)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:825)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:1058)
	at org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:1156)
	at org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:422)
	at org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:183)
	at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:226)
	at org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:908)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:1453)
	at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:131)
	at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.connect(PhoenixEmbeddedDriver.java:112)
	at sqlline.SqlLine$DatabaseConnection.connect(SqlLine.java:4650)
	at sqlline.SqlLine$DatabaseConnection.getConnection(SqlLine.java:4701)
	at sqlline.SqlLine$Commands.connect(SqlLine.java:3942)
	at sqlline.SqlLine$Commands.connect(SqlLine.java:3851)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at sqlline.SqlLine$ReflectiveCommandHandler.execute(SqlLine.java:2810)
	at sqlline.SqlLine.dispatch(SqlLine.java:817)
	at sqlline.SqlLine.initArgs(SqlLine.java:633)
	at sqlline.SqlLine.begin(SqlLine.java:680)
	at sqlline.SqlLine.mainWithInputRedirection(SqlLine.java:441)
	at sqlline.SqlLine.main(SqlLine.java:424)
Caused by: org.apache.hadoop.hbase.TableNotEnabledException: SYSTEM.CATALOG is disabled.
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1094)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1084)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:904)
	at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:78)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:120)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:97)
	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:90)
	at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:51)
	at org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService$Stub.getVersion(MetaDataProtos.java:6805)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl$3.call(ConnectionQueryServicesImpl.java:909)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl$3.call(ConnectionQueryServicesImpl.java:901)
	at org.apache.hadoop.hbase.client.HTable$16.call(HTable.java:1586)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. PHOENIX-1346 Base exception gets wrapped as incompatible jar exception

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Activity

            People

              samarthjain Samarth Jain
              cartershanklin Carter Shanklin
              Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: