[FLINK-25749] YARNSessionFIFOSecuredITCase.testDetachedMode fails on AZP - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Critical
Resolution: Fixed
Affects Version/s: 1.15.0
Fix Version/s: None
Component/s: Deployment / YARN
Labels:
- test-stability

Description

The test YARNSessionFIFOSecuredITCase.testDetachedMode fails on AZP:

2022-01-21T03:28:18.3712993Z Jan 21 03:28:18 java.lang.AssertionError: 
2022-01-21T03:28:18.3715115Z Jan 21 03:28:18 Found a file /__w/2/s/flink-yarn-tests/target/flink-yarn-tests-fifo-secured/flink-yarn-tests-fifo-secured-logDir-nm-0_0/application_1642735639007_0002/container_1642735639007_0002_01_000001/jobmanager.log with a prohibited string (one of [Exception, Started SelectChannelConnector@0.0.0.0:8081]). Excerpts:
2022-01-21T03:28:18.3716389Z Jan 21 03:28:18 [
2022-01-21T03:28:18.3717531Z Jan 21 03:28:18 2022-01-21 03:27:56,921 INFO  org.apache.flink.runtime.resourcemanager.ResourceManagerServiceImpl [] - Resource manager service is not running. Ignore revoking leadership.
2022-01-21T03:28:18.3720496Z Jan 21 03:28:18 2022-01-21 03:27:56,922 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Stopped dispatcher akka.tcp://flink@11c5f741db81:37697/user/rpc/dispatcher_0.
2022-01-21T03:28:18.3722401Z Jan 21 03:28:18 2022-01-21 03:27:56,922 INFO  org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl [] - Interrupted while waiting for queue
2022-01-21T03:28:18.3723661Z Jan 21 03:28:18 java.lang.InterruptedException: null
2022-01-21T03:28:18.3724529Z Jan 21 03:28:18 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_292]
2022-01-21T03:28:18.3725450Z Jan 21 03:28:18 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_292]
2022-01-21T03:28:18.3726239Z Jan 21 03:28:18 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) ~[?:1.8.0_292]
2022-01-21T03:28:18.3727618Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:323) [hadoop-yarn-client-2.8.5.jar:?]
2022-01-21T03:28:18.3729147Z Jan 21 03:28:18 2022-01-21 03:27:56,927 WARN  org.apache.hadoop.ipc.Client                                 [] - Failed to connect to server: 11c5f741db81/172.25.0.2:39121: retries get failed due to exceeded maximum allowed retries number: 0
2022-01-21T03:28:18.3730293Z Jan 21 03:28:18 java.nio.channels.ClosedByInterruptException: null
2022-01-21T03:28:18.3730834Z Jan 21 03:28:18 java.nio.channels.ClosedByInterruptException: null
2022-01-21T03:28:18.3731499Z Jan 21 03:28:18 	at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) ~[?:1.8.0_292]
2022-01-21T03:28:18.3732203Z Jan 21 03:28:18 	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:658) ~[?:1.8.0_292]
2022-01-21T03:28:18.3733478Z Jan 21 03:28:18 	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192) ~[hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3734470Z Jan 21 03:28:18 	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) ~[hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3735432Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:685) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3736414Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:788) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3737734Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:410) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3738853Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1550) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3739752Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.Client.call(Client.java:1381) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3740638Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.Client.call(Client.java:1345) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3741589Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3742621Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3743549Z Jan 21 03:28:18 	at com.sun.proxy.$Proxy51.stopContainers(Unknown Source) [?:?]
2022-01-21T03:28:18.3744684Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.stopContainers(ContainerManagementProtocolPBClientImpl.java:120) [hadoop-yarn-common-2.8.5.jar:?]
2022-01-21T03:28:18.3745594Z Jan 21 03:28:18 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_292]
2022-01-21T03:28:18.3746221Z Jan 21 03:28:18 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_292]
2022-01-21T03:28:18.3746937Z Jan 21 03:28:18 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_292]
2022-01-21T03:28:18.3747615Z Jan 21 03:28:18 	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_292]
2022-01-21T03:28:18.3748595Z Jan 21 03:28:18 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3749706Z Jan 21 03:28:18 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3750820Z Jan 21 03:28:18 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3751915Z Jan 21 03:28:18 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3753193Z Jan 21 03:28:18 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3753988Z Jan 21 03:28:18 	at com.sun.proxy.$Proxy52.stopContainers(Unknown Source) [?:?]
2022-01-21T03:28:18.3754973Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.client.api.impl.NMClientImpl.stopContainerInternal(NMClientImpl.java:316) [hadoop-yarn-client-2.8.5.jar:?]
2022-01-21T03:28:18.3756059Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.client.api.impl.NMClientImpl.stopContainer(NMClientImpl.java:271) [hadoop-yarn-client-2.8.5.jar:?]
2022-01-21T03:28:18.3757457Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl$StatefulContainer$StopContainerTransition.transition(NMClientAsyncImpl.java:541) [hadoop-yarn-client-2.8.5.jar:?]
2022-01-21T03:28:18.3758840Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl$StatefulContainer$StopContainerTransition.transition(NMClientAsyncImpl.java:532) [hadoop-yarn-client-2.8.5.jar:?]
2022-01-21T03:28:18.3760149Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385) [hadoop-yarn-common-2.8.5.jar:?]
2022-01-21T03:28:18.3761466Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) [hadoop-yarn-common-2.8.5.jar:?]
2022-01-21T03:28:18.3762578Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) [hadoop-yarn-common-2.8.5.jar:?]
2022-01-21T03:28:18.3763987Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) [hadoop-yarn-common-2.8.5.jar:?]
2022-01-21T03:28:18.3765348Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl$StatefulContainer.handle(NMClientAsyncImpl.java:617) [hadoop-yarn-client-2.8.5.jar:?]
2022-01-21T03:28:18.3766590Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl$ContainerEventProcessor.run(NMClientAsyncImpl.java:676) [hadoop-yarn-client-2.8.5.jar:?]
2022-01-21T03:28:18.3767443Z Jan 21 03:28:18 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292]
2022-01-21T03:28:18.3768153Z Jan 21 03:28:18 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292]
2022-01-21T03:28:18.3768778Z Jan 21 03:28:18 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
2022-01-21T03:28:18.3769835Z Jan 21 03:28:18 2022-01-21 03:27:56,930 WARN  org.apache.flink.yarn.YarnResourceManagerDriver              [] - Error while calling YARN Node Manager to stop container container_1642735639007_0002_01_000002.
2022-01-21T03:28:18.3770910Z Jan 21 03:28:18 java.io.IOException: Failed on local exception: java.nio.channels.ClosedByInterruptException; Host Details : local host is: "11c5f741db81/172.25.0.2"; destination host is: "11c5f741db81":39121; 
2022-01-21T03:28:18.3772073Z Jan 21 03:28:18 	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:782) ~[hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3773188Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1493) ~[hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3774186Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.Client.call(Client.java:1435) ~[hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3775072Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.Client.call(Client.java:1345) ~[hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3776041Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) ~[hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3777094Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) ~[hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3777769Z Jan 21 03:28:18 	at com.sun.proxy.$Proxy51.stopContainers(Unknown Source) ~[?:?]
2022-01-21T03:28:18.3778869Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.stopContainers(ContainerManagementProtocolPBClientImpl.java:120) ~[hadoop-yarn-common-2.8.5.jar:?]
2022-01-21T03:28:18.3779720Z Jan 21 03:28:18 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_292]
2022-01-21T03:28:18.3780182Z Jan 21 03:28:18 ]
2022-01-21T03:28:18.3780584Z Jan 21 03:28:18 	at org.junit.Assert.fail(Assert.java:89)
2022-01-21T03:28:18.3781169Z Jan 21 03:28:18 	at org.apache.flink.yarn.YarnTestBase.ensureNoProhibitedStringInLogFiles(YarnTestBase.java:591)
2022-01-21T03:28:18.3782025Z Jan 21 03:28:18 	at org.apache.flink.yarn.YARNSessionFIFOITCase.checkForProhibitedLogContents(YARNSessionFIFOITCase.java:82)
2022-01-21T03:28:18.3782677Z Jan 21 03:28:18 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2022-01-21T03:28:18.3783557Z Jan 21 03:28:18 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2022-01-21T03:28:18.3784250Z Jan 21 03:28:18 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2022-01-21T03:28:18.3784878Z Jan 21 03:28:18 	at java.lang.reflect.Method.invoke(Method.java:498)
2022-01-21T03:28:18.3785487Z Jan 21 03:28:18 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
2022-01-21T03:28:18.3786168Z Jan 21 03:28:18 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2022-01-21T03:28:18.3786840Z Jan 21 03:28:18 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
2022-01-21T03:28:18.3787505Z Jan 21 03:28:18 	at org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46)
2022-01-21T03:28:18.3788173Z Jan 21 03:28:18 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
2022-01-21T03:28:18.3788909Z Jan 21 03:28:18 	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
2022-01-21T03:28:18.3789557Z Jan 21 03:28:18 	at org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
2022-01-21T03:28:18.3790179Z Jan 21 03:28:18 	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
2022-01-21T03:28:18.3790755Z Jan 21 03:28:18 	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
2022-01-21T03:28:18.3791399Z Jan 21 03:28:18 	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
2022-01-21T03:28:18.3792038Z Jan 21 03:28:18 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
2022-01-21T03:28:18.3792667Z Jan 21 03:28:18 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
2022-01-21T03:28:18.3793646Z Jan 21 03:28:18 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
2022-01-21T03:28:18.3794282Z Jan 21 03:28:18 	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
2022-01-21T03:28:18.3794889Z Jan 21 03:28:18 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
2022-01-21T03:28:18.3795483Z Jan 21 03:28:18 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
2022-01-21T03:28:18.3796096Z Jan 21 03:28:18 	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
2022-01-21T03:28:18.3796697Z Jan 21 03:28:18 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
2022-01-21T03:28:18.3797324Z Jan 21 03:28:18 	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
2022-01-21T03:28:18.3797967Z Jan 21 03:28:18 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
2022-01-21T03:28:18.3798600Z Jan 21 03:28:18 	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
2022-01-21T03:28:18.3799223Z Jan 21 03:28:18 	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
2022-01-21T03:28:18.3799806Z Jan 21 03:28:18 	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
2022-01-21T03:28:18.3800367Z Jan 21 03:28:18 	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
2022-01-21T03:28:18.3800963Z Jan 21 03:28:18 	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
2022-01-21T03:28:18.3801528Z Jan 21 03:28:18 	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
2022-01-21T03:28:18.3802074Z Jan 21 03:28:18 	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
2022-01-21T03:28:18.3802689Z Jan 21 03:28:18 	at org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
2022-01-21T03:28:18.3803675Z Jan 21 03:28:18 	at org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
2022-01-21T03:28:18.3804487Z Jan 21 03:28:18 	at org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
2022-01-21T03:28:18.3805205Z Jan 21 03:28:18 	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
2022-01-21T03:28:18.3805990Z Jan 21 03:28:18 	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
2022-01-21T03:28:18.3806794Z Jan 21 03:28:18 	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
2022-01-21T03:28:18.3807616Z Jan 21 03:28:18 	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
2022-01-21T03:28:18.3808420Z Jan 21 03:28:18 	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
2022-01-21T03:28:18.3809106Z Jan 21 03:28:18 	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
2022-01-21T03:28:18.3809759Z Jan 21 03:28:18 	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
2022-01-21T03:28:18.3810463Z Jan 21 03:28:18 	at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
2022-01-21T03:28:18.3835763Z Jan 21 03:28:18 	at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
2022-01-21T03:28:18.3836503Z Jan 21 03:28:18 	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:188)
2022-01-21T03:28:18.3837248Z Jan 21 03:28:18 	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154)
2022-01-21T03:28:18.3838212Z Jan 21 03:28:18 	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
2022-01-21T03:28:18.3838953Z Jan 21 03:28:18 	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
2022-01-21T03:28:18.3839649Z Jan 21 03:28:18 	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
2022-01-21T03:28:18.3840279Z Jan 21 03:28:18 	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
2022-01-21T03:28:18.3840921Z Jan 21 03:28:18 	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)

The test YARNSessionFIFOSecuredITCase.testQueryCluster failed with:

2022-01-21T03:28:18.3842564Z Jan 21 03:28:18 java.lang.AssertionError: 
2022-01-21T03:28:18.3844597Z Jan 21 03:28:18 Found a file /__w/2/s/flink-yarn-tests/target/flink-yarn-tests-fifo-secured/flink-yarn-tests-fifo-secured-logDir-nm-0_0/application_1642735639007_0002/container_1642735639007_0002_01_000001/jobmanager.log with a prohibited string (one of [Exception, Started SelectChannelConnector@0.0.0.0:8081]). Excerpts:
2022-01-21T03:28:18.3845483Z Jan 21 03:28:18 [
2022-01-21T03:28:18.3846666Z Jan 21 03:28:18 2022-01-21 03:27:56,921 INFO  org.apache.flink.runtime.resourcemanager.ResourceManagerServiceImpl [] - Resource manager service is not running. Ignore revoking leadership.
2022-01-21T03:28:18.3848108Z Jan 21 03:28:18 2022-01-21 03:27:56,922 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Stopped dispatcher akka.tcp://flink@11c5f741db81:37697/user/rpc/dispatcher_0.
2022-01-21T03:28:18.3849225Z Jan 21 03:28:18 2022-01-21 03:27:56,922 INFO  org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl [] - Interrupted while waiting for queue
2022-01-21T03:28:18.3849824Z Jan 21 03:28:18 java.lang.InterruptedException: null
2022-01-21T03:28:18.3850480Z Jan 21 03:28:18 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_292]
2022-01-21T03:28:18.3851311Z Jan 21 03:28:18 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_292]
2022-01-21T03:28:18.3852206Z Jan 21 03:28:18 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) ~[?:1.8.0_292]
2022-01-21T03:28:18.3853638Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:323) [hadoop-yarn-client-2.8.5.jar:?]
2022-01-21T03:28:18.3855047Z Jan 21 03:28:18 2022-01-21 03:27:56,927 WARN  org.apache.hadoop.ipc.Client                                 [] - Failed to connect to server: 11c5f741db81/172.25.0.2:39121: retries get failed due to exceeded maximum allowed retries number: 0
2022-01-21T03:28:18.3855925Z Jan 21 03:28:18 java.nio.channels.ClosedByInterruptException: null
2022-01-21T03:28:18.3856444Z Jan 21 03:28:18 java.nio.channels.ClosedByInterruptException: null
2022-01-21T03:28:18.3857075Z Jan 21 03:28:18 	at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) ~[?:1.8.0_292]
2022-01-21T03:28:18.3857778Z Jan 21 03:28:18 	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:658) ~[?:1.8.0_292]
2022-01-21T03:28:18.3858858Z Jan 21 03:28:18 	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192) ~[hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3859778Z Jan 21 03:28:18 	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) ~[hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3860702Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:685) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3861692Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:788) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3862939Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:410) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3863944Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1550) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3864820Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.Client.call(Client.java:1381) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3865660Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.Client.call(Client.java:1345) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3866604Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3867586Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3868233Z Jan 21 03:28:18 	at com.sun.proxy.$Proxy51.stopContainers(Unknown Source) [?:?]
2022-01-21T03:28:18.3869278Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.stopContainers(ContainerManagementProtocolPBClientImpl.java:120) [hadoop-yarn-common-2.8.5.jar:?]
2022-01-21T03:28:18.3870212Z Jan 21 03:28:18 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_292]
2022-01-21T03:28:18.3870850Z Jan 21 03:28:18 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_292]
2022-01-21T03:28:18.3871580Z Jan 21 03:28:18 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_292]
2022-01-21T03:28:18.3872230Z Jan 21 03:28:18 	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_292]
2022-01-21T03:28:18.3873375Z Jan 21 03:28:18 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3874477Z Jan 21 03:28:18 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3875544Z Jan 21 03:28:18 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3876609Z Jan 21 03:28:18 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3877809Z Jan 21 03:28:18 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346) [hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3878470Z Jan 21 03:28:18 	at com.sun.proxy.$Proxy52.stopContainers(Unknown Source) [?:?]
2022-01-21T03:28:18.3879381Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.client.api.impl.NMClientImpl.stopContainerInternal(NMClientImpl.java:316) [hadoop-yarn-client-2.8.5.jar:?]
2022-01-21T03:28:18.3880442Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.client.api.impl.NMClientImpl.stopContainer(NMClientImpl.java:271) [hadoop-yarn-client-2.8.5.jar:?]
2022-01-21T03:28:18.3881623Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl$StatefulContainer$StopContainerTransition.transition(NMClientAsyncImpl.java:541) [hadoop-yarn-client-2.8.5.jar:?]
2022-01-21T03:28:18.3883111Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl$StatefulContainer$StopContainerTransition.transition(NMClientAsyncImpl.java:532) [hadoop-yarn-client-2.8.5.jar:?]
2022-01-21T03:28:18.3884553Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385) [hadoop-yarn-common-2.8.5.jar:?]
2022-01-21T03:28:18.3885610Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) [hadoop-yarn-common-2.8.5.jar:?]
2022-01-21T03:28:18.3886619Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) [hadoop-yarn-common-2.8.5.jar:?]
2022-01-21T03:28:18.3887692Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) [hadoop-yarn-common-2.8.5.jar:?]
2022-01-21T03:28:18.3888827Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl$StatefulContainer.handle(NMClientAsyncImpl.java:617) [hadoop-yarn-client-2.8.5.jar:?]
2022-01-21T03:28:18.3890014Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl$ContainerEventProcessor.run(NMClientAsyncImpl.java:676) [hadoop-yarn-client-2.8.5.jar:?]
2022-01-21T03:28:18.3890841Z Jan 21 03:28:18 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292]
2022-01-21T03:28:18.3891528Z Jan 21 03:28:18 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292]
2022-01-21T03:28:18.3892129Z Jan 21 03:28:18 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
2022-01-21T03:28:18.3893275Z Jan 21 03:28:18 2022-01-21 03:27:56,930 WARN  org.apache.flink.yarn.YarnResourceManagerDriver              [] - Error while calling YARN Node Manager to stop container container_1642735639007_0002_01_000002.
2022-01-21T03:28:18.3894308Z Jan 21 03:28:18 java.io.IOException: Failed on local exception: java.nio.channels.ClosedByInterruptException; Host Details : local host is: "11c5f741db81/172.25.0.2"; destination host is: "11c5f741db81":39121; 
2022-01-21T03:28:18.3895436Z Jan 21 03:28:18 	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:782) ~[hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3896356Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1493) ~[hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3897214Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.Client.call(Client.java:1435) ~[hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3898060Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.Client.call(Client.java:1345) ~[hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3898994Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) ~[hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3899987Z Jan 21 03:28:18 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) ~[hadoop-common-2.8.5.jar:?]
2022-01-21T03:28:18.3900639Z Jan 21 03:28:18 	at com.sun.proxy.$Proxy51.stopContainers(Unknown Source) ~[?:?]
2022-01-21T03:28:18.3901798Z Jan 21 03:28:18 	at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.stopContainers(ContainerManagementProtocolPBClientImpl.java:120) ~[hadoop-yarn-common-2.8.5.jar:?]
2022-01-21T03:28:18.3902618Z Jan 21 03:28:18 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_292]
2022-01-21T03:28:18.3903137Z Jan 21 03:28:18 ]
2022-01-21T03:28:18.3903592Z Jan 21 03:28:18 	at org.junit.Assert.fail(Assert.java:89)
2022-01-21T03:28:18.3904183Z Jan 21 03:28:18 	at org.apache.flink.yarn.YarnTestBase.ensureNoProhibitedStringInLogFiles(YarnTestBase.java:591)
2022-01-21T03:28:18.3904901Z Jan 21 03:28:18 	at org.apache.flink.yarn.YARNSessionFIFOITCase.checkForProhibitedLogContents(YARNSessionFIFOITCase.java:82)
2022-01-21T03:28:18.3905526Z Jan 21 03:28:18 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2022-01-21T03:28:18.3906105Z Jan 21 03:28:18 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2022-01-21T03:28:18.3906769Z Jan 21 03:28:18 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2022-01-21T03:28:18.3907469Z Jan 21 03:28:18 	at java.lang.reflect.Method.invoke(Method.java:498)
2022-01-21T03:28:18.3908059Z Jan 21 03:28:18 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
2022-01-21T03:28:18.3908700Z Jan 21 03:28:18 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2022-01-21T03:28:18.3909350Z Jan 21 03:28:18 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
2022-01-21T03:28:18.3909993Z Jan 21 03:28:18 	at org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46)
2022-01-21T03:28:18.3910632Z Jan 21 03:28:18 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
2022-01-21T03:28:18.3911243Z Jan 21 03:28:18 	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
2022-01-21T03:28:18.3911869Z Jan 21 03:28:18 	at org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
2022-01-21T03:28:18.3912465Z Jan 21 03:28:18 	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
2022-01-21T03:28:18.3913092Z Jan 21 03:28:18 	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
2022-01-21T03:28:18.3913775Z Jan 21 03:28:18 	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
2022-01-21T03:28:18.3914387Z Jan 21 03:28:18 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
2022-01-21T03:28:18.3914995Z Jan 21 03:28:18 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
2022-01-21T03:28:18.3915645Z Jan 21 03:28:18 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
2022-01-21T03:28:18.3916252Z Jan 21 03:28:18 	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
2022-01-21T03:28:18.3916823Z Jan 21 03:28:18 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
2022-01-21T03:28:18.3917405Z Jan 21 03:28:18 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
2022-01-21T03:28:18.3917976Z Jan 21 03:28:18 	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
2022-01-21T03:28:18.3918554Z Jan 21 03:28:18 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
2022-01-21T03:28:18.3919152Z Jan 21 03:28:18 	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
2022-01-21T03:28:18.3919778Z Jan 21 03:28:18 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
2022-01-21T03:28:18.3920386Z Jan 21 03:28:18 	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
2022-01-21T03:28:18.3920980Z Jan 21 03:28:18 	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
2022-01-21T03:28:18.3921545Z Jan 21 03:28:18 	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
2022-01-21T03:28:18.3922166Z Jan 21 03:28:18 	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
2022-01-21T03:28:18.3922727Z Jan 21 03:28:18 	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
2022-01-21T03:28:18.3923398Z Jan 21 03:28:18 	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
2022-01-21T03:28:18.3923921Z Jan 21 03:28:18 	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
2022-01-21T03:28:18.3924505Z Jan 21 03:28:18 	at org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
2022-01-21T03:28:18.3925163Z Jan 21 03:28:18 	at org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
2022-01-21T03:28:18.3926090Z Jan 21 03:28:18 	at org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
2022-01-21T03:28:18.3926927Z Jan 21 03:28:18 	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
2022-01-21T03:28:18.3927700Z Jan 21 03:28:18 	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
2022-01-21T03:28:18.3928487Z Jan 21 03:28:18 	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
2022-01-21T03:28:18.3929388Z Jan 21 03:28:18 	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
2022-01-21T03:28:18.3930162Z Jan 21 03:28:18 	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
2022-01-21T03:28:18.3930858Z Jan 21 03:28:18 	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
2022-01-21T03:28:18.3931520Z Jan 21 03:28:18 	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
2022-01-21T03:28:18.3932245Z Jan 21 03:28:18 	at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
2022-01-21T03:28:18.3933105Z Jan 21 03:28:18 	at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
2022-01-21T03:28:18.3933889Z Jan 21 03:28:18 	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:188)
2022-01-21T03:28:18.3934645Z Jan 21 03:28:18 	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154)
2022-01-21T03:28:18.3935380Z Jan 21 03:28:18 	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
2022-01-21T03:28:18.3936073Z Jan 21 03:28:18 	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
2022-01-21T03:28:18.3936725Z Jan 21 03:28:18 	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
2022-01-21T03:28:18.3937348Z Jan 21 03:28:18 	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=29841&view=logs&j=fc5181b0-e452-5c8f-68de-1097947f6483&t=995c650b-6573-581c-9ce6-7ad4cc038461

Attachments

Issue Links

is fixed by

FLINK-25277 Introduce explicit shutdown signalling between TaskManager and JobManager

Resolved

Activity

People

Assignee:: Till Rohrmann

Reporter:: Till Rohrmann

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 21/Jan/22 09:10

Updated:: 24/Jan/22 08:53

Resolved:: 24/Jan/22 08:53