Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2134

AsyncHbase Sink bugfix plus tests errors on Windows

    XMLWordPrintableJSON

Details

    • Fixes for AsyncHbase sink to run on Windows

    Description

      Problem appears to be inabilty to handle windows paths.

      13/07/23 17:16:35 INFO server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x1400e08d22b0000 type:delete cxid:0x13 zxid:0xb txntype:-1 reqpath:n/a Error Path:/hbase/backup-masters/192.168.50.128,55884,1374624994459 Error:KeeperErrorCode = NoNode for /hbase/backup-masters/192.168.50.128,55884,1374624994459
      13/07/23 17:16:35 WARN zookeeper.RecoverableZooKeeper: Node /hbase/backup-masters/192.168.50.128,55884,1374624994459 already deleted, and this is not a retry
      13/07/23 17:16:35 INFO master.ActiveMasterManager: Master=192.168.50.128,55884,1374624994459
      13/07/23 17:16:35 ERROR master.HMaster: Unhandled exception. Starting shutdown.
      java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file://C:%5CUsers%5CROSHAN~1%5CAppData%5CLocal%5CTemp%5C1374624993647-0%5Chbase.tmp
      	at org.apache.hadoop.fs.Path.initialize(Path.java:148)
      	at org.apache.hadoop.fs.Path.<init>(Path.java:71)
      	at org.apache.hadoop.fs.Path.<init>(Path.java:50)
      	at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:111)
      	at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:532)
      	at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:391)
      	at java.lang.Thread.run(Thread.java:722)
      Caused by: java.net.URISyntaxException: Relative path in absolute URI: file://C:%5CUsers%5CROSHAN~1%5CAppData%5CLocal%5CTemp%5C1374624993647-0%5Chbase.tmp
      	at java.net.URI.checkPath(URI.java:1804)
      	at java.net.URI.<init>(URI.java:752)
      	at org.apache.hadoop.fs.Path.initialize(Path.java:145)
      	... 6 more
      13/07/23 17:16:35 INFO master.HMaster: Aborting
      13/07/23 17:16:35 INFO ipc.HBaseServer: Stopping server on 55884
      13/07/23 17:16:35 INFO ipc.HBaseServer: Stopping IPC Server listener on 55884
      13/07/23 17:16:35 INFO ipc.HBaseServer: Stopping IPC Server Responder
      13/07/23 17:16:35 INFO ipc.HBaseServer: Stopping IPC Server Responder
      13/07/23 17:16:35 INFO ipc.HBaseServer: IPC Server handler 0 on 55884: exiting
      13/07/23 17:16:35 INFO ipc.HBaseServer: IPC Server handler 1 on 55884: exiting
      13/07/23 17:16:35 INFO ipc.HBaseServer: IPC Server handler 2 on 55884: exiting
      13/07/23 17:16:35 INFO ipc.HBaseServer: IPC Server handler 3 on 55884: exiting
      13/07/23 17:16:35 INFO ipc.HBaseServer: IPC Server handler 4 on 55884: exiting
      13/07/23 17:16:35 INFO ipc.HBaseServer: IPC Server handler 5 on 55884: exiting
      13/07/23 17:16:35 INFO ipc.HBaseServer: IPC Server handler 6 on 55884: exiting
      13/07/23 17:16:35 INFO ipc.HBaseServer: IPC Server handler 7 on 55884: exiting
      13/07/23 17:16:35 INFO ipc.HBaseServer: IPC Server handler 8 on 55884: exiting
      13/07/23 17:16:35 INFO ipc.HBaseServer: IPC Server handler 9 on 55884: exiting
      13/07/23 17:16:35 INFO ipc.HBaseServer: REPL IPC Server handler 0 on 55884: exiting
      13/07/23 17:16:35 INFO ipc.HBaseServer: REPL IPC Server handler 1 on 55884: exiting
      13/07/23 17:16:35 INFO ipc.HBaseServer: REPL IPC Server handler 2 on 55884: exiting
      13/07/23 17:16:35 INFO server.PrepRequestProcessor: Processed session termination for sessionid: 0x1400e08d22b0000
      13/07/23 17:16:35 INFO server.NIOServerCnxn: Closed socket connection for client /192.168.50.128:55911 which had sessionid 0x1400e08d22b0000
      13/07/23 17:16:35 INFO zookeeper.ZooKeeper: Session: 0x1400e08d22b0000 closed
      13/07/23 17:16:35 INFO master.HMaster: HMaster main thread exiting
      13/07/23 17:16:35 INFO zookeeper.ClientCnxn: EventThread shut down
      13/07/23 17:16:35 INFO regionserver.ShutdownHook: Installed shutdown hook thread: Shutdownhook:RegionServer:0;192.168.50.128,55912,1374624994999
      13/07/23 17:16:35 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=0.0.0.0:2181 sessionTimeout=180000 watcher=regionserver:55912
      13/07/23 17:16:35 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 5824@WIN-6AKNOG7MFK5
      13/07/23 17:16:35 INFO zookeeper.ClientCnxn: Opening socket connection to server 0.0.0.0/0.0.0.0:2181. Will not attempt to authenticate using SASL (unknown error)
      13/07/23 17:16:35 INFO server.NIOServerCnxnFactory: Accepted socket connection from /192.168.50.128:55939
      13/07/23 17:16:35 INFO zookeeper.ClientCnxn: Socket connection established to 0.0.0.0/0.0.0.0:2181, initiating session
      13/07/23 17:16:35 INFO server.ZooKeeperServer: Client attempting to establish new session at /192.168.50.128:55939
      13/07/23 17:16:35 INFO zookeeper.ClientCnxn: Session establishment complete on server 0.0.0.0/0.0.0.0:2181, sessionid = 0x1400e08d22b0001, negotiated timeout = 40000
      13/07/23 17:16:35 INFO server.ZooKeeperServer: Established session 0x1400e08d22b0001 with negotiated timeout 40000 for client /192.168.50.128:55939
      13/07/23 17:19:55 ERROR hbase.MiniHBaseCluster: Error starting cluster
      java.lang.RuntimeException: Master not initialized after 200 seconds
      	at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:206)
      	at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:420)
      	at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:216)
      	at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:86)
      	at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:77)
      	at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:64)
      	at org.apache.flume.sink.hbase.TestAsyncHBaseSink.setUp(TestAsyncHBaseSink.java:123)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:601)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
      	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
      	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
      	at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
      	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:77)
      	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:195)
      	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
      13/07/23 17:19:55 WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/shutdown
      13/07/23 17:19:55 INFO util.RetryCounter: Sleeping 2000ms before retry #1...
      13/07/23 17:19:57 WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/shutdown
      13/07/23 17:19:57 INFO util.RetryCounter: Sleeping 4000ms before retry #2...
      13/07/23 17:20:01 WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/shutdown
      13/07/23 17:20:01 INFO util.RetryCounter: Sleeping 8000ms before retry #3...
      13/07/23 17:20:09 WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/shutdown
      13/07/23 17:20:09 ERROR zookeeper.RecoverableZooKeeper: ZooKeeper delete failed after 3 retries
      13/07/23 17:20:09 ERROR master.HMaster: ZooKeeper exception trying to set cluster as down in ZK
      org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/shutdown
      	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
      	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
      	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
      	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:133)
      	at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1197)
      	at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1186)
      	at org.apache.hadoop.hbase.zookeeper.ClusterStatusTracker.setClusterDown(ClusterStatusTracker.java:80)
      	at org.apache.hadoop.hbase.master.HMaster.shutdown(HMaster.java:1672)
      	at org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:234)
      	at org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:427)
      	at org.apache.hadoop.hbase.MiniHBaseCluster.shutdown(MiniHBaseCluster.java:492)
      	at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:222)
      	at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:86)
      	at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:77)
      	at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:64)
      	at org.apache.flume.sink.hbase.TestAsyncHBaseSink.setUp(TestAsyncHBaseSink.java:123)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:601)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
      	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
      	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
      	at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
      	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:77)
      	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:195)
      	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
      13/07/23 17:20:09 INFO regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-19,5,main]
      13/07/23 17:20:09 ERROR server.NIOServerCnxnFactory: Thread Thread[Shutdownhook:RegionServer:0;192.168.50.128,55912,1374624994999,5,main] died
      java.lang.NullPointerException
      	at org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1810)
      	at org.apache.hadoop.hbase.regionserver.ShutdownHook$ShutdownHookThread.run(ShutdownHook.java:113)
      
      java.lang.NullPointerException
      	at org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1810)
      	at org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:241)
      	at org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:427)
      	at org.apache.hadoop.hbase.MiniHBaseCluster.shutdown(MiniHBaseCluster.java:492)
      	at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:222)
      	at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:86)
      	at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:77)
      	at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:64)
      	at org.apache.flume.sink.hbase.TestAsyncHBaseSink.setUp(TestAsyncHBaseSink.java:123)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
      	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
      	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
      	at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
      	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
      
      
      java.lang.NullPointerException
      	at org.apache.flume.sink.hbase.TestAsyncHBaseSink.tearDown(TestAsyncHBaseSink.java:162)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
      	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
      	at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
      	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
      
      
      java.lang.NullPointerException
      	at org.apache.flume.sink.hbase.TestAsyncHBaseSink.tearDown(TestAsyncHBaseSink.java:162)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
      	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
      	at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
      	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
      
      
      java.lang.NullPointerException
      	at org.apache.flume.sink.hbase.TestAsyncHBaseSink.tearDown(TestAsyncHBaseSink.java:162)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
      	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
      	at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
      	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
      
      
      java.lang.NullPointerException
      	at org.apache.flume.sink.hbase.TestAsyncHBaseSink.tearDown(TestAsyncHBaseSink.java:162)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
      	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
      	at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
      	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
      
      Disconnected from the target VM, address: '127.0.0.1:55877', transport: 'socket'
      
      Process finished with exit code -1
      

      Attachments

        1. FLUME-2134.patch
          6 kB
          Roshan Naik
        2. FLUME-2134.v2.patch
          6 kB
          Roshan Naik
        3. FLUME-2134.v3.patch
          7 kB
          Roshan Naik

        Issue Links

          Activity

            People

              roshan_naik Roshan Naik
              roshan_naik Roshan Naik
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: