Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-1604

Concurrent write issue on hdfs when deploy coprocessor

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: v1.5.1
    • Fix Version/s: v1.5.2
    • Component/s: Job Engine
    • Labels:
      None

      Description

      When multiple cubing jobs reach the Create HTable Job Step at same time, they have the change to upload coprocessor with same name, which cause a write conflict.
      Here's the error log:
      2016-04-21 18:05:29,930 INFO [pool-9-thread-9] steps.CubeHTableUtil (CubeHTableUtil.java:createHTable(106)) - hbase will use gzip to compress data
      2016-04-21 18:05:29,932 INFO [pool-9-thread-10] steps.CubeHTableUtil (CubeHTableUtil.java:createHTable(106)) - hbase will use gzip to compress data
      2016-04-21 18:05:29,992 WARN [Thread-117] hdfs.DFSClient (DFSOutputStream.java:run(558)) - DataStreamer Exception
      org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /kylin/kylin_metadata/coprocessor/kylin-storage-hbase-1.5.1-coprocessor-5.jar (inode 341288): File does not exist. [Lease. Holder: DFSClient_NONMAPREDUCE_-492525792_1, pendingcreates: 1]
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3439)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:3242)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3080)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3040)
      at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:789)
      at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
      at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:415)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)

      at org.apache.hadoop.ipc.Client.call(Client.java:1476)
      at org.apache.hadoop.ipc.Client.call(Client.java:1407)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
      at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
      at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418)
      at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
      at com.sun.proxy.$Proxy10.addBlock(Unknown Source)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1430)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1226)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
      2016-04-21 18:05:29,994 ERROR [pool-9-thread-9] util.DeployCoprocessorCLI (DeployCoprocessorCLI.java:deployCoprocessor(170)) - Error deploying coprocessor on KYLIN_7DJ3R4X2OC
      org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /kylin/kylin_metadata/coprocessor/kylin-storage-hbase-1.5.1-coprocessor-5.jar (inode 341288): File does not exist. [Lease. Holder: DFSClient_NONMAPREDUCE_-492525792_1, pendingcreates: 1]
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3439)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:3242)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3080)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3040)
      at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:789)
      at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
      at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:415)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)

      at org.apache.hadoop.ipc.Client.call(Client.java:1476)
      at org.apache.hadoop.ipc.Client.call(Client.java:1407)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
      at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
      at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418)
      at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
      at com.sun.proxy.$Proxy10.addBlock(Unknown Source)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1430)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1226)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
      2016-04-21 18:05:29,995 ERROR [pool-9-thread-9] util.DeployCoprocessorCLI (DeployCoprocessorCLI.java:deployCoprocessor(171)) - Will try creating the table without coprocessor.
      2016-04-21 18:05:30,120 INFO [pool-9-thread-10] util.DeployCoprocessorCLI (DeployCoprocessorCLI.java:addCoprocessorOnHTable(187)) - Add coprocessor on KYLIN_5E5NJHHSBZ
      2016-04-21 18:05:30,122 INFO [pool-9-thread-10] util.DeployCoprocessorCLI (DeployCoprocessorCLI.java:deployCoprocessor(167)) - hbase table KYLIN_5E5NJHHSBZ deployed with coprocessor.

        Attachments

          Activity

            People

            • Assignee:
              lidong_sjtu Dong Li
              Reporter:
              lidong_sjtu Dong Li
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: