Uploaded image for project: 'Geode'
  1. Geode
  2. GEODE-9618

CI Failure: HdelDUnitTest fails with RedisCommandExecutionException ERR

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.15.0
    • redis
    • None

    Description

      org.apache.geode.redis.internal.executor.hash.HdelDUnitTest > testConcurrentHdel_whenServerCrashesAndRestarts FAILED
          java.lang.RuntimeException: java.util.concurrent.ExecutionException: io.lettuce.core.RedisCommandExecutionException: ERR The server had an internal error please try again
              at org.apache.geode.redis.ConcurrentLoopingThreads.await(ConcurrentLoopingThreads.java:78)
              at org.apache.geode.redis.internal.executor.hash.HdelDUnitTest.testConcurrentHdel_whenServerCrashesAndRestarts(HdelDUnitTest.java:137)
      
              Caused by:
              java.util.concurrent.ExecutionException: io.lettuce.core.RedisCommandExecutionException: ERR The server had an internal error please try again
                  at java.util.concurrent.FutureTask.report(FutureTask.java:122)
                  at java.util.concurrent.FutureTask.get(FutureTask.java:206)
                  at org.apache.geode.redis.ConcurrentLoopingThreads.await(ConcurrentLoopingThreads.java:74)
                  ... 1 more
      
                  Caused by:
                  io.lettuce.core.RedisCommandExecutionException: ERR The server had an internal error please try again
                      at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:137)
                      at io.lettuce.core.internal.Exceptions.bubble(Exceptions.java:72)
                      at io.lettuce.core.internal.Futures.awaitOrCancel(Futures.java:250)
                      at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:130)
                      at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
                      at com.sun.proxy.$Proxy50.hdel(Unknown Source)
                      at org.apache.geode.redis.internal.executor.hash.HdelDUnitTest.lambda$null$2(HdelDUnitTest.java:130)
                      at org.apache.geode.redis.internal.executor.hash.HdelDUnitTest.retryableCommand(HdelDUnitTest.java:146)
                      at org.apache.geode.redis.internal.executor.hash.HdelDUnitTest.lambda$testConcurrentHdel_whenServerCrashesAndRestarts$3(HdelDUnitTest.java:130)
      
                      Caused by:
                      io.lettuce.core.RedisCommandExecutionException: ERR The server had an internal error please try again
                          at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:137)
                          at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:110)
                          at io.lettuce.core.protocol.AsyncCommand.completeResult(AsyncCommand.java:120)
                          at io.lettuce.core.protocol.AsyncCommand.complete(AsyncCommand.java:111)
                          at io.lettuce.core.protocol.CommandWrapper.complete(CommandWrapper.java:63)
                          at io.lettuce.core.cluster.ClusterCommand.complete(ClusterCommand.java:65)
                          at io.lettuce.core.protocol.CommandWrapper.complete(CommandWrapper.java:63)
                          at io.lettuce.core.protocol.CommandHandler.complete(CommandHandler.java:746)
                          at io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:681)
                          at io.lettuce.core.protocol.CommandHandler.channelRead(CommandHandler.java:598)
                          at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
                          at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
                          at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
                          at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
                          at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
                          at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
                          at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
                          at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
                          at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
                          at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
                          at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
                          at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
                          at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
                          at java.lang.Thread.run(Thread.java:748)
      

      There are also quite a few suspect string failures:

      org.apache.geode.redis.internal.executor.hash.HdelDUnitTest > classMethod FAILED
          java.lang.AssertionError: Suspicious strings were written to the log during this run.
          Fix the strings or use IgnoredException.addIgnoredException to ignore.
          -----------------------------------------------------------------------
          Found suspect string in 'dunit_suspect-vm1.log' at line 1592
      
          [error 2021/09/18 06:54:51.201 UTC <GeodeRedisServer-WorkerThread-5> tid=102] GeodeRedisServer-Unexpected error handler for [id: 0x70109872, L:/127.0.0.1:45693 - R:/127.0.0.1:52868]
          org.apache.geode.internal.serialization.DSFIDNotFoundException: Unknown DataSerializableFixedID: 160
            at org.apache.geode.internal.serialization.internal.DSFIDSerializerImpl.create(DSFIDSerializerImpl.java:383)
            at org.apache.geode.internal.DSFIDFactory.create(DSFIDFactory.java:1027)
            at org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2517)
            at org.apache.geode.DataSerializer.readObject(DataSerializer.java:2864)
            at org.apache.geode.internal.cache.UpdateOperation$UpdateMessage.fromData(UpdateOperation.java:366)
            at org.apache.geode.internal.serialization.internal.DSFIDSerializerImpl.invokeFromData(DSFIDSerializerImpl.java:317)
            at org.apache.geode.internal.serialization.internal.DSFIDSerializerImpl.create(DSFIDSerializerImpl.java:368)
            at org.apache.geode.internal.DSFIDFactory.create(DSFIDFactory.java:1027)
            at org.apache.geode.internal.InternalDataSerializer.readDSFID(InternalDataSerializer.java:2396)
            at org.apache.geode.internal.InternalDataSerializer.readDSFID(InternalDataSerializer.java:2410)
            at org.apache.geode.internal.tcp.Connection.readMessage(Connection.java:3017)
            at org.apache.geode.internal.tcp.Connection.processInputBuffer(Connection.java:2809)
            at org.apache.geode.internal.tcp.Connection.readMessages(Connection.java:1656)
            at org.apache.geode.internal.tcp.Connection.run(Connection.java:1496)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)
      
          -----------------------------------------------------------------------
          Found suspect string in 'dunit_suspect-vm1.log' at line 1703
      
          [error 2021/09/18 06:54:51.215 UTC <GeodeRedisServer-WorkerThread-5> tid=102] GeodeRedisServer-Unexpected error handler for [id: 0x70109872, L:/127.0.0.1:45693 - R:/127.0.0.1:52868]
          org.apache.geode.internal.serialization.DSFIDNotFoundException: Unknown DataSerializableFixedID: 160
            at org.apache.geode.internal.serialization.internal.DSFIDSerializerImpl.create(DSFIDSerializerImpl.java:383)
            at org.apache.geode.internal.DSFIDFactory.create(DSFIDFactory.java:1027)
            at org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2517)
            at org.apache.geode.DataSerializer.readObject(DataSerializer.java:2864)
            at org.apache.geode.internal.cache.UpdateOperation$UpdateMessage.fromData(UpdateOperation.java:366)
            at org.apache.geode.internal.serialization.internal.DSFIDSerializerImpl.invokeFromData(DSFIDSerializerImpl.java:317)
            at org.apache.geode.internal.serialization.internal.DSFIDSerializerImpl.create(DSFIDSerializerImpl.java:368)
            at org.apache.geode.internal.DSFIDFactory.create(DSFIDFactory.java:1027)
            at org.apache.geode.internal.InternalDataSerializer.readDSFID(InternalDataSerializer.java:2396)
            at org.apache.geode.internal.InternalDataSerializer.readDSFID(InternalDataSerializer.java:2410)
            at org.apache.geode.internal.tcp.Connection.readMessage(Connection.java:3017)
            at org.apache.geode.internal.tcp.Connection.processInputBuffer(Connection.java:2809)
            at org.apache.geode.internal.tcp.Connection.readMessages(Connection.java:1656)
            at org.apache.geode.internal.tcp.Connection.run(Connection.java:1496)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)
      
          -----------------------------------------------------------------------
          Found suspect string in 'dunit_suspect-vm2.log' at line 1848
      
          [fatal 2021/09/18 06:54:51.156 UTC <P2P message reader for heavy-lifter-2fd170e2-351b-5128-b871-23599672bdc9(server-1:402190)<v1>:41834 unshared ordered sender uid=10 dom #1 local port=42508 remote port=41124> tid=63] Error deserializing message
          java.io.IOException: Could not create an instance of org.apache.geode.internal.cache.UpdateOperation$UpdateMessage .
            at org.apache.geode.internal.serialization.internal.DSFIDSerializerImpl.invokeFromData(DSFIDSerializerImpl.java:330)
            at org.apache.geode.internal.serialization.internal.DSFIDSerializerImpl.create(DSFIDSerializerImpl.java:368)
            at org.apache.geode.internal.DSFIDFactory.create(DSFIDFactory.java:1027)
            at org.apache.geode.internal.InternalDataSerializer.readDSFID(InternalDataSerializer.java:2396)
            at org.apache.geode.internal.InternalDataSerializer.readDSFID(InternalDataSerializer.java:2410)
            at org.apache.geode.internal.tcp.Connection.readMessage(Connection.java:3017)
            at org.apache.geode.internal.tcp.Connection.processInputBuffer(Connection.java:2809)
            at org.apache.geode.internal.tcp.Connection.readMessages(Connection.java:1656)
            at org.apache.geode.internal.tcp.Connection.run(Connection.java:1496)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)
          Caused by: org.apache.geode.internal.serialization.DSFIDNotFoundException: Unknown DataSerializableFixedID: 160
            at org.apache.geode.internal.serialization.internal.DSFIDSerializerImpl.create(DSFIDSerializerImpl.java:383)
            at org.apache.geode.internal.DSFIDFactory.create(DSFIDFactory.java:1027)
            at org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2517)
            at org.apache.geode.DataSerializer.readObject(DataSerializer.java:2864)
            at org.apache.geode.internal.cache.UpdateOperation$UpdateMessage.fromData(UpdateOperation.java:366)
            at org.apache.geode.internal.serialization.internal.DSFIDSerializerImpl.invokeFromData(DSFIDSerializerImpl.java:317)
            ... 11 more
      
          -----------------------------------------------------------------------
          Found suspect string in 'dunit_suspect-vm2.log' at line 1871
      
          [fatal 2021/09/18 06:54:51.209 UTC <P2P message reader for heavy-lifter-2fd170e2-351b-5128-b871-23599672bdc9(server-1:402190)<v1>:41834 unshared ordered sender uid=10 dom #1 local port=42508 remote port=41124> tid=63] Error deserializing message
          java.io.IOException: Could not create an instance of org.apache.geode.internal.cache.UpdateOperation$UpdateMessage .
            at org.apache.geode.internal.serialization.internal.DSFIDSerializerImpl.invokeFromData(DSFIDSerializerImpl.java:330)
            at org.apache.geode.internal.serialization.internal.DSFIDSerializerImpl.create(DSFIDSerializerImpl.java:368)
            at org.apache.geode.internal.DSFIDFactory.create(DSFIDFactory.java:1027)
            at org.apache.geode.internal.InternalDataSerializer.readDSFID(InternalDataSerializer.java:2396)
            at org.apache.geode.internal.InternalDataSerializer.readDSFID(InternalDataSerializer.java:2410)
            at org.apache.geode.internal.tcp.Connection.readMessage(Connection.java:3017)
            at org.apache.geode.internal.tcp.Connection.processInputBuffer(Connection.java:2809)
            at org.apache.geode.internal.tcp.Connection.readMessages(Connection.java:1656)
            at org.apache.geode.internal.tcp.Connection.run(Connection.java:1496)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)
          Caused by: org.apache.geode.internal.serialization.DSFIDNotFoundException: Unknown DataSerializableFixedID: 160
            at org.apache.geode.internal.serialization.internal.DSFIDSerializerImpl.create(DSFIDSerializerImpl.java:383)
            at org.apache.geode.internal.DSFIDFactory.create(DSFIDFactory.java:1027)
            at org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2517)
            at org.apache.geode.DataSerializer.readObject(DataSerializer.java:2864)
            at org.apache.geode.internal.cache.UpdateOperation$UpdateMessage.fromData(UpdateOperation.java:366)
            at org.apache.geode.internal.serialization.internal.DSFIDSerializerImpl.invokeFromData(DSFIDSerializerImpl.java:317)
            ... 11 more
              at org.junit.Assert.fail(Assert.java:89)
              at org.apache.geode.test.dunit.internal.DUnitLauncher.closeAndCheckForSuspects(DUnitLauncher.java:409)
              at org.apache.geode.test.dunit.internal.DUnitLauncher.closeAndCheckForSuspects(DUnitLauncher.java:425)
              at org.apache.geode.test.dunit.rules.ClusterStartupRule.after(ClusterStartupRule.java:186)
              at org.apache.geode.test.dunit.rules.ClusterStartupRule.access$100(ClusterStartupRule.java:70)
              at org.apache.geode.test.dunit.rules.ClusterStartupRule$1.evaluate(ClusterStartupRule.java:141)
              at org.apache.geode.test.junit.rules.serializable.SerializableExternalResource$1.evaluate(SerializableExternalResource.java:38)
              at org.junit.rules.RunRules.evaluate(RunRules.java:20)
              at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
              at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
              at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
              at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
              at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
              at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
              at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
              at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
              at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
              at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
              at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
              at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
              at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
              at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
              at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
              at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
              at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
              at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
              at java.lang.Thread.run(Thread.java:748)
      

      Attachments

        Activity

          People

            jens.deppe Jens Deppe
            klund Kirk Lund
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: