Details

    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      The default setting of the configuration parameter 'hbase.coprocessor.abortonerror' has been changed from "false" to "true". This will cause HBase processes to abort if a coprocessor cannot be loaded, fails to initialize, or throws an unexpected exception. If "false", this instead will allow the server to continue execution in such cases but the system wide state of the coprocessor in question will become inconsistent as it will be properly executing in only a subset of servers, so this is most useful for debugging a new coprocessor deployment on a test cluster. Running a secure production cluster with this setting set to "false" is not advised.
      Show
      The default setting of the configuration parameter 'hbase.coprocessor.abortonerror' has been changed from "false" to "true". This will cause HBase processes to abort if a coprocessor cannot be loaded, fails to initialize, or throws an unexpected exception. If "false", this instead will allow the server to continue execution in such cases but the system wide state of the coprocessor in question will become inconsistent as it will be properly executing in only a subset of servers, so this is most useful for debugging a new coprocessor deployment on a test cluster. Running a secure production cluster with this setting set to "false" is not advised.

      Description

      When registering a coprocessor with a missing dependency, the regionserver gets stuck in an infinite fail loop. Restarting the regionserver and/or master has no affect.

      E.g.,

      Load coprocessor from my-coproc.jar, that uses an external dependency (kafka) that is not included with HBase.

      12/09/24 13:13:15 INFO handler.OpenRegionHandler: Opening of region {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking as FAILED_OPEN in ZK
      12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
      12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
      12/09/24 13:13:15 INFO regionserver.HRegionServer: Received request to open region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
      12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
      12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
      12/09/24 13:13:15 DEBUG regionserver.HRegion: Opening region: {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
      12/09/24 13:13:15 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
      12/09/24 13:13:15 INFO coprocessor.CoprocessorHost: Class com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded from a file - file:/path/to/my-coproc.jar.
      12/09/24 13:13:16 ERROR handler.OpenRegionHandler: Failed open of region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting to roll back the global memstore size.
      java.lang.IllegalStateException: Could not instantiate a region instance.
      	at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
      	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
      	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
      	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
      	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:680)
      Caused by: java.lang.reflect.InvocationTargetException
      	at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      	at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
      	... 7 more
      Caused by: java.lang.NoClassDefFoundError: kafka/common/NoBrokersForPartitionException
      	at java.lang.Class.getDeclaredConstructors0(Native Method)
      	at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
      	at java.lang.Class.getConstructor0(Class.java:2699)
      	at java.lang.Class.newInstance0(Class.java:326)
      	at java.lang.Class.newInstance(Class.java:308)
      	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
      	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
      	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
      	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
      	at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
      	... 11 more
      Caused by: java.lang.ClassNotFoundException: kafka.common.NoBrokersForPartitionException
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      	... 21 more
      12/09/24 13:13:16 INFO handler.OpenRegionHandler: Opening of region {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking as FAILED_OPEN in ZK
      12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
      12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
      12/09/24 13:13:16 INFO regionserver.HRegionServer: Received request to open region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
      12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
      12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
      12/09/24 13:13:16 DEBUG regionserver.HRegion: Opening region: {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
      12/09/24 13:13:16 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
      12/09/24 13:13:16 INFO coprocessor.CoprocessorHost: Class com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded from a file - file:/path/to/my-coproc.jar.
      12/09/24 13:13:17 ERROR handler.OpenRegionHandler: Failed open of region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting to roll back the global memstore size.
      java.lang.IllegalStateException: Could not instantiate a region instance.
      	at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
      	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
      	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
      	at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
      	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:680)
      Caused by: java.lang.reflect.InvocationTargetException
      	at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      	at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
      	... 7 more
      Caused by: java.lang.NoClassDefFoundError: kafka/common/NoBrokersForPartitionException
      	at java.lang.Class.getDeclaredConstructors0(Native Method)
      	at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
      	at java.lang.Class.getConstructor0(Class.java:2699)
      	at java.lang.Class.newInstance0(Class.java:326)
      	at java.lang.Class.newInstance(Class.java:308)
      	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
      	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
      	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
      	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
      	at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
      	... 11 more
      Caused by: java.lang.ClassNotFoundException: kafka.common.NoBrokersForPartitionException
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      	... 21 more
      

      Ad infinitum.

      It seems that upon failing to open a region after adding a coprocessor, that coprocessor should be unregister or at least disabled/blacklisted.

      1. 6873.patch
        23 kB
        Andrew Purtell
      2. 6873.patch
        23 kB
        Andrew Purtell
      3. 6873.patch
        23 kB
        Andrew Purtell
      4. 6873.patch
        24 kB
        Andrew Purtell
      5. 6873.patch
        31 kB
        Andrew Purtell
      6. 6873.patch
        34 kB
        Andrew Purtell
      7. 6873.patch
        35 kB
        Andrew Purtell

        Issue Links

          Activity

          Hide
          enis Enis Soztutar added a comment -

          Closing this issue after 0.99.0 release.

          Show
          enis Enis Soztutar added a comment - Closing this issue after 0.99.0 release.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #55 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/55/)
          HBASE-6873. Clean up Coprocessor loading failure handling (apurtell: rev 1558869)

          • /hbase/trunk/hbase-common/src/main/resources/hbase-default.xml
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerCoprocessorHost.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCoprocessorHost.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/constraint/TestConstraint.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java
          • /hbase/trunk/hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #55 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/55/ ) HBASE-6873 . Clean up Coprocessor loading failure handling (apurtell: rev 1558869) /hbase/trunk/hbase-common/src/main/resources/hbase-default.xml /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerCoprocessorHost.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCoprocessorHost.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/constraint/TestConstraint.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java /hbase/trunk/hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #78 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/78/)
          HBASE-6873. Clean up Coprocessor loading failure handling (apurtell: rev 1558870)

          • /hbase/branches/0.98/hbase-common/src/main/resources/hbase-default.xml
          • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
          • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
          • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
          • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerCoprocessorHost.java
          • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCoprocessorHost.java
          • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/constraint/TestConstraint.java
          • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java
          • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
          • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java
          • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java
          • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java
          • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java
          • /hbase/branches/0.98/hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #78 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/78/ ) HBASE-6873 . Clean up Coprocessor loading failure handling (apurtell: rev 1558870) /hbase/branches/0.98/hbase-common/src/main/resources/hbase-default.xml /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerCoprocessorHost.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCoprocessorHost.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/constraint/TestConstraint.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java /hbase/branches/0.98/hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in HBase-0.98 #86 (See https://builds.apache.org/job/HBase-0.98/86/)
          HBASE-6873. Clean up Coprocessor loading failure handling (apurtell: rev 1558870)

          • /hbase/branches/0.98/hbase-common/src/main/resources/hbase-default.xml
          • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
          • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
          • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
          • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerCoprocessorHost.java
          • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCoprocessorHost.java
          • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/constraint/TestConstraint.java
          • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java
          • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
          • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java
          • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java
          • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java
          • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java
          • /hbase/branches/0.98/hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in HBase-0.98 #86 (See https://builds.apache.org/job/HBase-0.98/86/ ) HBASE-6873 . Clean up Coprocessor loading failure handling (apurtell: rev 1558870) /hbase/branches/0.98/hbase-common/src/main/resources/hbase-default.xml /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerCoprocessorHost.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCoprocessorHost.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/constraint/TestConstraint.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java /hbase/branches/0.98/hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java
          Hide
          apurtell Andrew Purtell added a comment - - edited

          I checked the build report of HBase-TRUNK #4825 (https://builds.apache.org/job/HBase-TRUNK/4825/) . This is an occasional failure with TestHBaseFsck that turns up now and again in some environments, generally builds.apache.org, and is not related.

          Show
          apurtell Andrew Purtell added a comment - - edited I checked the build report of HBase-TRUNK #4825 ( https://builds.apache.org/job/HBase-TRUNK/4825/ ) . This is an occasional failure with TestHBaseFsck that turns up now and again in some environments, generally builds.apache.org, and is not related.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in HBase-TRUNK #4825 (See https://builds.apache.org/job/HBase-TRUNK/4825/)
          HBASE-6873. Clean up Coprocessor loading failure handling (apurtell: rev 1558869)

          • /hbase/trunk/hbase-common/src/main/resources/hbase-default.xml
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerCoprocessorHost.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCoprocessorHost.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/constraint/TestConstraint.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java
          • /hbase/trunk/hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #4825 (See https://builds.apache.org/job/HBase-TRUNK/4825/ ) HBASE-6873 . Clean up Coprocessor loading failure handling (apurtell: rev 1558869) /hbase/trunk/hbase-common/src/main/resources/hbase-default.xml /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerCoprocessorHost.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCoprocessorHost.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/constraint/TestConstraint.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java /hbase/trunk/hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java
          Hide
          apurtell Andrew Purtell added a comment -

          I ran the full test suite (trunk, with this patch applied) locally on two boxes 25 times and no tests failed except TestAssignmentManagerOnCluster, which is HBASE-10341.

          Committed to trunk and 0.98 branch. Thanks everyone for the reviews and good discussion.

          Show
          apurtell Andrew Purtell added a comment - I ran the full test suite (trunk, with this patch applied) locally on two boxes 25 times and no tests failed except TestAssignmentManagerOnCluster, which is HBASE-10341 . Committed to trunk and 0.98 branch. Thanks everyone for the reviews and good discussion.
          Hide
          apurtell Andrew Purtell added a comment -

          Great, will commit in a little while to trunk and 0.98.

          Show
          apurtell Andrew Purtell added a comment - Great, will commit in a little while to trunk and 0.98.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12623067/6873.patch
          against trunk revision .
          ATTACHMENT ID: 12623067

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 30 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 site. The patch appears to cause mvn site goal to fail.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8433//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8433//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8433//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8433//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8433//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8433//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8433//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8433//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8433//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8433//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8433//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12623067/6873.patch against trunk revision . ATTACHMENT ID: 12623067 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 30 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8433//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8433//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8433//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8433//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8433//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8433//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8433//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8433//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8433//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8433//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8433//console This message is automatically generated.
          Hide
          apurtell Andrew Purtell added a comment -

          The TestShell failure was real:

          2014-01-14 21:56:49,890 DEBUG [RS_OPEN_REGION-acer:48810-1] coprocessor.CoprocessorHost(189): Loading coprocessor class SimpleRegionObserver with path hdfs:/foo.jar and priority 12
          2014-01-14 21:56:49,896 ERROR [RS_OPEN_REGION-acer:48810-1] coprocessor.CoprocessorHost(771): The coprocessor SimpleRegionObserver threw an unexpected exception
          java.io.FileNotFoundException: File does not exist: hdfs:/foo.jar
          

          One of the shell tests wants to set a fake coprocessor attribute to prove that it can do it.

          Attaching new patch.

          Show
          apurtell Andrew Purtell added a comment - The TestShell failure was real: 2014-01-14 21:56:49,890 DEBUG [RS_OPEN_REGION-acer:48810-1] coprocessor.CoprocessorHost(189): Loading coprocessor class SimpleRegionObserver with path hdfs:/foo.jar and priority 12 2014-01-14 21:56:49,896 ERROR [RS_OPEN_REGION-acer:48810-1] coprocessor.CoprocessorHost(771): The coprocessor SimpleRegionObserver threw an unexpected exception java.io.FileNotFoundException: File does not exist: hdfs:/foo.jar One of the shell tests wants to set a fake coprocessor attribute to prove that it can do it. Attaching new patch.
          Hide
          apurtell Andrew Purtell added a comment -

          Found it, HBASE-9804

          Show
          apurtell Andrew Purtell added a comment - Found it, HBASE-9804
          Hide
          ghelmling Gary Helmling added a comment -

          I was just looking for an issue I filed a while back, an option for pausing the master after the deployment of system tables but before deployment of user tables, but can't find it offhand. If meta is online it should be possible to alter the HTD/HCDs.

          Yeah, that would be a good remediation option here and for other cases as well. Constraining the failure domain to only the table the coprocessor is configured on would still be nice, but as you point out there is potential for other abuse that wouldn't be constrained anyway.

          Show
          ghelmling Gary Helmling added a comment - I was just looking for an issue I filed a while back, an option for pausing the master after the deployment of system tables but before deployment of user tables, but can't find it offhand. If meta is online it should be possible to alter the HTD/HCDs. Yeah, that would be a good remediation option here and for other cases as well. Constraining the failure domain to only the table the coprocessor is configured on would still be nice, but as you point out there is potential for other abuse that wouldn't be constrained anyway.
          Hide
          apurtell Andrew Purtell added a comment -

          Yeah but at least here someone is following along.

          That doesn't quite work for coprocessors configured as table attributes (more like wipe the table dir from HDFS),

          I was just looking for an issue I filed a while back, an option for pausing the master after the deployment of system tables but before deployment of user tables, but can't find it offhand. If meta is online it should be possible to alter the HTD/HCDs.

          Show
          apurtell Andrew Purtell added a comment - Yeah but at least here someone is following along. That doesn't quite work for coprocessors configured as table attributes (more like wipe the table dir from HDFS), I was just looking for an issue I filed a while back, an option for pausing the master after the deployment of system tables but before deployment of user tables, but can't find it offhand. If meta is online it should be possible to alter the HTD/HCDs.
          Hide
          ghelmling Gary Helmling added a comment -

          Site file edit and rolling restart, currently. Something better we should tackle in a follow on issue.

          That doesn't quite work for coprocessors configured as table attributes (more like wipe the table dir from HDFS), but I've sidelined this issue enough. Let's move discussion to a follow on issue.

          Show
          ghelmling Gary Helmling added a comment - Site file edit and rolling restart, currently. Something better we should tackle in a follow on issue. That doesn't quite work for coprocessors configured as table attributes (more like wipe the table dir from HDFS), but I've sidelined this issue enough. Let's move discussion to a follow on issue.
          Hide
          apurtell Andrew Purtell added a comment -

          I'd just like to make sure we have an easy operational way of reversing the problem.

          Site file edit and rolling restart, currently. Something better we should tackle in a follow on issue.

          Show
          apurtell Andrew Purtell added a comment - I'd just like to make sure we have an easy operational way of reversing the problem. Site file edit and rolling restart, currently. Something better we should tackle in a follow on issue.
          Hide
          apurtell Andrew Purtell added a comment -

          Just to follow up on that, we handle the security implications of coprocessors being a way to inject arbitrary code into the runtime by restricting who can do it. If the AccessController is installed, only users with CREATE or ADMIN privilege can set up table coprocessors. Only the superuser, effectively, can install system coprocessors, she who edits the hbase-site.xml file.

          Show
          apurtell Andrew Purtell added a comment - Just to follow up on that, we handle the security implications of coprocessors being a way to inject arbitrary code into the runtime by restricting who can do it. If the AccessController is installed, only users with CREATE or ADMIN privilege can set up table coprocessors. Only the superuser, effectively, can install system coprocessors, she who edits the hbase-site.xml file.
          Hide
          ghelmling Gary Helmling added a comment -

          We are certainly totally vulnerable to misbehaving coprocessors until we do better isolation through an externalized process or whatever.

          What I'm more worried about with the config default change is rendering a cluster inoperable due to a misconfiguration of a coprocessor. For example:

          • create a table with a table-level coprocessor as an attribute
          • the table coprocessor when loaded has an unresolvable dependency (as described here)
          • so the first table region bounces from host to host taking down regionservers, failing to open

          Now we have a .tableinfo sitting around with the coprocessor configured in it, but we can't bring the cluster back up to alter the table definition and remove the CP or replace it with a CP config pointing to a new jar with the required dependency. Maybe I'm missing some easy way of resolving this (hopefully I am), but that seems nasty. Whether here or in a follow on enhancement, I'd just like to make sure we have an easy operational way of reversing the problem.

          Though I agree that correctness (which we're currently sacrificing with hbase.coprocessor.abortonerror=false) is equally or more important.

          Show
          ghelmling Gary Helmling added a comment - We are certainly totally vulnerable to misbehaving coprocessors until we do better isolation through an externalized process or whatever. What I'm more worried about with the config default change is rendering a cluster inoperable due to a misconfiguration of a coprocessor. For example: create a table with a table-level coprocessor as an attribute the table coprocessor when loaded has an unresolvable dependency (as described here) so the first table region bounces from host to host taking down regionservers, failing to open Now we have a .tableinfo sitting around with the coprocessor configured in it, but we can't bring the cluster back up to alter the table definition and remove the CP or replace it with a CP config pointing to a new jar with the required dependency. Maybe I'm missing some easy way of resolving this (hopefully I am), but that seems nasty. Whether here or in a follow on enhancement, I'd just like to make sure we have an easy operational way of reversing the problem. Though I agree that correctness (which we're currently sacrificing with hbase.coprocessor.abortonerror=false) is equally or more important.
          Hide
          stack stack added a comment -

          Let's not forget from the beginning coprocessors install Java code into the RS without any process or address space isolation.

          Smile.

          Show
          stack stack added a comment - Let's not forget from the beginning coprocessors install Java code into the RS without any process or address space isolation. Smile.
          Hide
          apurtell Andrew Purtell added a comment -

          I don't think that TestShell failure is related but I will look into it. Be back shortly.

          Show
          apurtell Andrew Purtell added a comment - I don't think that TestShell failure is related but I will look into it. Be back shortly.
          Hide
          apurtell Andrew Purtell added a comment - - edited

          I think a CP taking down the cluster something worthy of discussion, yes... another issue.

          Hang on guys.

          blah blah prePut (blah blah) {
             for (;;) { }
          }
          

          Now I've (eventually) taken the cluster down by jamming up all RPC workers. Let's not forget from the beginning coprocessors install Java code into the RS without any process or address space isolation.

          The fix for this issue is HBASE-4047

          Show
          apurtell Andrew Purtell added a comment - - edited I think a CP taking down the cluster something worthy of discussion, yes... another issue. Hang on guys. blah blah prePut (blah blah) { for (;;) { } } Now I've (eventually) taken the cluster down by jamming up all RPC workers. Let's not forget from the beginning coprocessors install Java code into the RS without any process or address space isolation. The fix for this issue is HBASE-4047
          Hide
          stack stack added a comment -

          I think a CP taking down the cluster something worthy of discussion, yes... another issue.

          Show
          stack stack added a comment - I think a CP taking down the cluster something worthy of discussion, yes... another issue.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12622996/6873.patch
          against trunk revision .
          ATTACHMENT ID: 12622996

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 26 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 site. The patch appears to cause mvn site goal to fail.

          -1 core tests. The patch failed these unit tests:

          -1 core zombie tests. There are 1 zombie test(s): at org.apache.hadoop.hbase.client.TestShell.testRunShellTests(TestShell.java:70)

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8426//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8426//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12622996/6873.patch against trunk revision . ATTACHMENT ID: 12622996 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 26 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: -1 core zombie tests . There are 1 zombie test(s): at org.apache.hadoop.hbase.client.TestShell.testRunShellTests(TestShell.java:70) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8426//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8426//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8426//console This message is automatically generated.
          Hide
          apurtell Andrew Purtell added a comment -

          Waiting for HadoopQA to come back, will commit if good. Thanks for the review stack

          Show
          apurtell Andrew Purtell added a comment - Waiting for HadoopQA to come back, will commit if good. Thanks for the review stack
          Hide
          apurtell Andrew Purtell added a comment -

          offline the table due to the coprocessor loading error (say with a prominent message on the master UI), but keep servers up so that the problem can be fixed

          A patch that does this would be great. Let's follow up on another JIRA.

          Show
          apurtell Andrew Purtell added a comment - offline the table due to the coprocessor loading error (say with a prominent message on the master UI), but keep servers up so that the problem can be fixed A patch that does this would be great. Let's follow up on another JIRA.
          Hide
          apurtell Andrew Purtell added a comment -

          But does this make it so that a misconfigured table-level coprocessor can bring down an entire cluster due to a missing dependency class?

          At this stage, that is the point, unless you change the configuration away from the new default to avoid that situation (previous behavior). Otherwise we ship coprocessors like the AccessController and VisibilityController that should fail close but instead they would fail open, which is unacceptable as default config.

          Show
          apurtell Andrew Purtell added a comment - But does this make it so that a misconfigured table-level coprocessor can bring down an entire cluster due to a missing dependency class? At this stage, that is the point, unless you change the configuration away from the new default to avoid that situation (previous behavior). Otherwise we ship coprocessors like the AccessController and VisibilityController that should fail close but instead they would fail open, which is unacceptable as default config.
          Hide
          ghelmling Gary Helmling added a comment -

          I understand the argument about data consistency in the case of unloaded coprocessors, so I see the need to change the default value of hbase.coprocessor.abortonerror. But does this make it so that a misconfigured table-level coprocessor can bring down an entire cluster due to a missing dependency class? If so, that really doesn't help our stability or multi-tenant story. Seems like a more ideal approach would be to offline the table due to the coprocessor loading error (say with a prominent message on the master UI), but keep servers up so that the problem can be fixed. For system level coprocessors, an abort might still be appropriate.

          Maybe out of scope for this change, but what do you think about this as a follow on issue?

          Show
          ghelmling Gary Helmling added a comment - I understand the argument about data consistency in the case of unloaded coprocessors, so I see the need to change the default value of hbase.coprocessor.abortonerror. But does this make it so that a misconfigured table-level coprocessor can bring down an entire cluster due to a missing dependency class? If so, that really doesn't help our stability or multi-tenant story. Seems like a more ideal approach would be to offline the table due to the coprocessor loading error (say with a prominent message on the master UI), but keep servers up so that the problem can be fixed. For system level coprocessors, an abort might still be appropriate. Maybe out of scope for this change, but what do you think about this as a follow on issue?
          Hide
          stack stack added a comment -

          lgtm Andrew Purtell It is nice and tidy.

          Show
          stack stack added a comment - lgtm Andrew Purtell It is nice and tidy.
          Hide
          apurtell Andrew Purtell added a comment -

          New patch addressing zombie test.

          Show
          apurtell Andrew Purtell added a comment - New patch addressing zombie test.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12622960/6873.patch
          against trunk revision .
          ATTACHMENT ID: 12622960

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 22 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 site. The patch appears to cause mvn site goal to fail.

          -1 core tests. The patch failed these unit tests:

          -1 core zombie tests. There are 1 zombie test(s): at org.apache.hadoop.hbase.constraint.TestConstraint.testIsUnloaded(TestConstraint.java:221)

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8422//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8422//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12622960/6873.patch against trunk revision . ATTACHMENT ID: 12622960 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 22 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: -1 core zombie tests . There are 1 zombie test(s): at org.apache.hadoop.hbase.constraint.TestConstraint.testIsUnloaded(TestConstraint.java:221) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8422//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8422//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8422//console This message is automatically generated.
          Hide
          apurtell Andrew Purtell added a comment -

          New patch

          Show
          apurtell Andrew Purtell added a comment - New patch
          Hide
          apurtell Andrew Purtell added a comment -

          The last changed kicked loose tests that were apparently skipped. Looks like I can't throw this change up without looking at every test first. Back in a bit.

          Show
          apurtell Andrew Purtell added a comment - The last changed kicked loose tests that were apparently skipped. Looks like I can't throw this change up without looking at every test first. Back in a bit.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12622807/6873.patch
          against trunk revision .
          ATTACHMENT ID: 12622807

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 11 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 site. The patch appears to cause mvn site goal to fail.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove
          org.apache.hadoop.hbase.coprocessor.TestClassLoading
          org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove

          -1 core zombie tests. There are 1 zombie test(s): at org.apache.hadoop.hbase.constraint.TestConstraint.testIsUnloaded(TestConstraint.java:221)

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8417//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8417//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12622807/6873.patch against trunk revision . ATTACHMENT ID: 12622807 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 11 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove org.apache.hadoop.hbase.coprocessor.TestClassLoading org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove -1 core zombie tests . There are 1 zombie test(s): at org.apache.hadoop.hbase.constraint.TestConstraint.testIsUnloaded(TestConstraint.java:221) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8417//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8417//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8417//console This message is automatically generated.
          Hide
          apurtell Andrew Purtell added a comment -

          With test fix for TestCoprocessorInterface, which relies on the old default.

          Show
          apurtell Andrew Purtell added a comment - With test fix for TestCoprocessorInterface, which relies on the old default.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12622793/6873.patch
          against trunk revision .
          ATTACHMENT ID: 12622793

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 site. The patch appears to cause mvn site goal to fail.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.coprocessor.TestCoprocessorInterface

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8415//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8415//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12622793/6873.patch against trunk revision . ATTACHMENT ID: 12622793 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 7 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.coprocessor.TestCoprocessorInterface Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8415//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8415//console This message is automatically generated.
          Hide
          apurtell Andrew Purtell added a comment -

          Again without Configuration#unset, which apparently is not in Hadoop 1.1

          Show
          apurtell Andrew Purtell added a comment - Again without Configuration#unset, which apparently is not in Hadoop 1.1
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12622772/6873.patch
          against trunk revision .
          ATTACHMENT ID: 12622772

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified tests.

          -1 hadoop1.0. The patch failed to compile against the hadoop 1.0 profile.
          Here is snippet of errors:

          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile (default-testCompile) on project hbase-server: Compilation failure
          [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java:[56,8] cannot find symbol
          [ERROR] symbol  : method unset(java.lang.String)
          [ERROR] location: class org.apache.hadoop.conf.Configuration
          [ERROR] -> [Help 1]
          org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile (default-testCompile) on project hbase-server: Compilation failure
          /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java:[56,8] cannot find symbol
          symbol  : method unset(java.lang.String)
          location: class org.apache.hadoop.conf.Configuration
          
          	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
          --
          Caused by: org.apache.maven.plugin.CompilationFailureException: Compilation failure
          /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java:[56,8] cannot find symbol
          symbol  : method unset(java.lang.String)
          location: class org.apache.hadoop.conf.Configuration
          
          	at org.apache.maven.plugin.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:729)

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8414//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12622772/6873.patch against trunk revision . ATTACHMENT ID: 12622772 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 7 new or modified tests. -1 hadoop1.0 . The patch failed to compile against the hadoop 1.0 profile. Here is snippet of errors: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile ( default -testCompile) on project hbase-server: Compilation failure [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java:[56,8] cannot find symbol [ERROR] symbol : method unset(java.lang. String ) [ERROR] location: class org.apache.hadoop.conf.Configuration [ERROR] -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile ( default -testCompile) on project hbase-server: Compilation failure /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java:[56,8] cannot find symbol symbol : method unset(java.lang. String ) location: class org.apache.hadoop.conf.Configuration at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213) -- Caused by: org.apache.maven.plugin.CompilationFailureException: Compilation failure /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java:[56,8] cannot find symbol symbol : method unset(java.lang. String ) location: class org.apache.hadoop.conf.Configuration at org.apache.maven.plugin.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:729) Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8414//console This message is automatically generated.
          Hide
          apurtell Andrew Purtell added a comment -

          Fix a silly issue with the test (configuration sharing)

          Show
          apurtell Andrew Purtell added a comment - Fix a silly issue with the test (configuration sharing)
          Hide
          apurtell Andrew Purtell added a comment -

          With new test.

          Show
          apurtell Andrew Purtell added a comment - With new test.
          Hide
          apurtell Andrew Purtell added a comment -

          Update hbase-default.xml also.

          Show
          apurtell Andrew Purtell added a comment - Update hbase-default.xml also.
          Hide
          apurtell Andrew Purtell added a comment -

          Here is what I'm thinking. Not tested yet, needs a unit test. Back with an update shortly.

          Show
          apurtell Andrew Purtell added a comment - Here is what I'm thinking. Not tested yet, needs a unit test. Back with an update shortly.
          Hide
          apurtell Andrew Purtell added a comment -

          Making this a blocker for 0.98, but specifically only:

          • If system coprocessors fail to load, just call abort().
          • If table coprocessors fail to load, call abort() unless explicitly told to ignore failures for that particular coprocessor (only in RegionCoprocessorHost).
          Show
          apurtell Andrew Purtell added a comment - Making this a blocker for 0.98, but specifically only: If system coprocessors fail to load, just call abort(). If table coprocessors fail to load, call abort() unless explicitly told to ignore failures for that particular coprocessor (only in RegionCoprocessorHost).
          Hide
          apurtell Andrew Purtell added a comment -

          Ted Yu Let's iterate on this after HBASE-7205 goes in.

          Show
          apurtell Andrew Purtell added a comment - Ted Yu Let's iterate on this after HBASE-7205 goes in.
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -

          @Andy:
          Do you want to attach latest patch for Hadoop QA to run tests ?

          Thanks

          Show
          yuzhihong@gmail.com Ted Yu added a comment - @Andy: Do you want to attach latest patch for Hadoop QA to run tests ? Thanks
          Hide
          apurtell Andrew Purtell added a comment -
          Show
          apurtell Andrew Purtell added a comment - I put up https://reviews.apache.org/r/7242/
          Hide
          apurtell Andrew Purtell added a comment -

          Or, for system CPs, since all region opens will fail, perhaps it's best to just abort the RS.

          Show
          apurtell Andrew Purtell added a comment - Or, for system CPs, since all region opens will fail, perhaps it's best to just abort the RS.
          Hide
          apurtell Andrew Purtell added a comment -

          Elliott Clark I was just pondering that very question. The first step is to clean up the above exception into an IOE with a more helpful message. The next is to extend CP loading slightly to allow region open to continue of a CP load fails, but not as the default behavior, it would have to be specified somehow.

          Haven't quite gotten to that "somehow". For table CPs it could be a tag in the table attribute defining the CP. For system CPs, looks like I have to introduce CP definition attributes there?

          Show
          apurtell Andrew Purtell added a comment - Elliott Clark I was just pondering that very question. The first step is to clean up the above exception into an IOE with a more helpful message. The next is to extend CP loading slightly to allow region open to continue of a CP load fails, but not as the default behavior, it would have to be specified somehow. Haven't quite gotten to that "somehow". For table CPs it could be a tag in the table attribute defining the CP. For system CPs, looks like I have to introduce CP definition attributes there?
          Hide
          eclark Elliott Clark added a comment -

          How does this work with the security co-processors. Not loading them is a case that the we want to fail hard.

          Show
          eclark Elliott Clark added a comment - How does this work with the security co-processors. Not loading them is a case that the we want to fail hard.
          Hide
          apurtell Andrew Purtell added a comment -

          Fair enough. Let me put up a patch.

          Show
          apurtell Andrew Purtell added a comment - Fair enough. Let me put up a patch.
          Hide
          mumrah David Arthur added a comment -

          At worst I'd think that a bad coprocessor jar would just make that coprocessor registration fail - not cripple the whole server.

          I'd argue that it's a bug because the regionserver is unrecoverable if this happens. Once it gets stuck in this failure loop, there's no way to get it unstuck (as far as I can see). So in effect, if you register a bad coprocessor, the whole server is unusable and unrecoverable.

          Show
          mumrah David Arthur added a comment - At worst I'd think that a bad coprocessor jar would just make that coprocessor registration fail - not cripple the whole server. I'd argue that it's a bug because the regionserver is unrecoverable if this happens. Once it gets stuck in this failure loop, there's no way to get it unstuck (as far as I can see). So in effect, if you register a bad coprocessor, the whole server is unusable and unrecoverable.
          Hide
          apurtell Andrew Purtell added a comment -

          Coprocessors are system level extensions. I'm not sure it's a bug if the deployment is faulty or corrupt and the regionserver refuses to start.

          Show
          apurtell Andrew Purtell added a comment - Coprocessors are system level extensions. I'm not sure it's a bug if the deployment is faulty or corrupt and the regionserver refuses to start.

            People

            • Assignee:
              apurtell Andrew Purtell
              Reporter:
              mumrah David Arthur
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development