Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Invalid
    • Affects Version/s: 0.90.1
    • Fix Version/s: 0.90.2
    • Component/s: None
    • Labels:
      None

      Description

      I inserted 100G or so of data and got a region stuck in a CLOSED state. Different region servers kept trying to open it, but they failed to transition OFFLINE->OPENING because it was in a CLOSED state.

      1. 6e2a39.log
        219 kB
        Todd Lipcon

        Activity

        Hide
        Todd Lipcon added a comment -

        err, this may be invalid... just found this in logs:

        hbase-todd-master-styx01.sf.cloudera.com.log-2011-02-16 12:04:53,905 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event RS_ZK_REGION_CLOSED
        hbase-todd-master-styx01.sf.cloudera.com.log-java.lang.NoClassDefFoundError: org/apache/zookeeper/proto/SetDataRequest
        hbase-todd-master-styx01.sf.cloudera.com.log- at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1031)
        hbase-todd-master-styx01.sf.cloudera.com.log- at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:708)
        hbase-todd-master-styx01.sf.cloudera.com.log- at org.apache.hadoop.hbase.zookeeper.ZKAssign.createOrForceNodeOffline(ZKAssign.java:248)
        hbase-todd-master-styx01.sf.cloudera.com.log- at org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:937)
        hbase-todd-master-styx01.sf.cloudera.com.log- at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:888)
        hbase-todd-master-styx01.sf.cloudera.com.log- at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:730)
        hbase-todd-master-styx01.sf.cloudera.com.log- at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:710)
        hbase-todd-master-styx01.sf.cloudera.com.log- at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:92)
        hbase-todd-master-styx01.sf.cloudera.com.log- at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
        hbase-todd-master-styx01.sf.cloudera.com.log- at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        hbase-todd-master-styx01.sf.cloudera.com.log- at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        hbase-todd-master-styx01.sf.cloudera.com.log- at java.lang.Thread.run(Thread.java:662)
        hbase-todd-master-styx01.sf.cloudera.com.log-Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.proto.SetDataRequest
        hbase-todd-master-styx01.sf.cloudera.com.log- at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        hbase-todd-master-styx01.sf.cloudera.com.log- at java.security.AccessController.doPrivileged(Native Method)
        hbase-todd-master-styx01.sf.cloudera.com.log- at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        hbase-todd-master-styx01.sf.cloudera.com.log- at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        hbase-todd-master-styx01.sf.cloudera.com.log- at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        hbase-todd-master-styx01.sf.cloudera.com.log- at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
        hbase-todd-master-styx01.sf.cloudera.com.log- ... 12 more

        I wonder if I did some maven command at that point which pwned my ZK jar or something.

        Show
        Todd Lipcon added a comment - err, this may be invalid... just found this in logs: hbase-todd-master-styx01.sf.cloudera.com.log-2011-02-16 12:04:53,905 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event RS_ZK_REGION_CLOSED hbase-todd-master-styx01.sf.cloudera.com.log-java.lang.NoClassDefFoundError: org/apache/zookeeper/proto/SetDataRequest hbase-todd-master-styx01.sf.cloudera.com.log- at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1031) hbase-todd-master-styx01.sf.cloudera.com.log- at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:708) hbase-todd-master-styx01.sf.cloudera.com.log- at org.apache.hadoop.hbase.zookeeper.ZKAssign.createOrForceNodeOffline(ZKAssign.java:248) hbase-todd-master-styx01.sf.cloudera.com.log- at org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:937) hbase-todd-master-styx01.sf.cloudera.com.log- at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:888) hbase-todd-master-styx01.sf.cloudera.com.log- at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:730) hbase-todd-master-styx01.sf.cloudera.com.log- at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:710) hbase-todd-master-styx01.sf.cloudera.com.log- at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:92) hbase-todd-master-styx01.sf.cloudera.com.log- at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151) hbase-todd-master-styx01.sf.cloudera.com.log- at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) hbase-todd-master-styx01.sf.cloudera.com.log- at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) hbase-todd-master-styx01.sf.cloudera.com.log- at java.lang.Thread.run(Thread.java:662) hbase-todd-master-styx01.sf.cloudera.com.log-Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.proto.SetDataRequest hbase-todd-master-styx01.sf.cloudera.com.log- at java.net.URLClassLoader$1.run(URLClassLoader.java:202) hbase-todd-master-styx01.sf.cloudera.com.log- at java.security.AccessController.doPrivileged(Native Method) hbase-todd-master-styx01.sf.cloudera.com.log- at java.net.URLClassLoader.findClass(URLClassLoader.java:190) hbase-todd-master-styx01.sf.cloudera.com.log- at java.lang.ClassLoader.loadClass(ClassLoader.java:307) hbase-todd-master-styx01.sf.cloudera.com.log- at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) hbase-todd-master-styx01.sf.cloudera.com.log- at java.lang.ClassLoader.loadClass(ClassLoader.java:248) hbase-todd-master-styx01.sf.cloudera.com.log- ... 12 more I wonder if I did some maven command at that point which pwned my ZK jar or something.

          People

          • Assignee:
            Unassigned
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development