[HDDS-7628] Intermittent failure in TestOzoneContainerWithTLS

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.4.0
• Component/s: test
• Labels: pull-request-available
• Target Version/s: 1.4.0
• Assignee: Sammi Chen
• Reporter: Attila Doroszlai
• Created: 12/Dec/22 19:42
• Updated: 27/Jun/23 20:15
• Resolved: 09/Jan/23 02:44

Description

Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 32.458 s <<< FAILURE! - in org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerWithTLS
TestOzoneContainerWithTLS.testContainerDownload  Time elapsed: 8.028 s  <<< ERROR!
java.io.IOException: 
java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:436)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.lambda$sendCommandWithTraceIDAndRetry$0(XceiverClientGrpc.java:349)
	at org.apache.hadoop.hdds.tracing.TracingUtil.executeInSpan(TracingUtil.java:177)
	at org.apache.hadoop.hdds.tracing.TracingUtil.executeInNewSpan(TracingUtil.java:151)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:343)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:265)
	at org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerWithTLS.createContainer(TestOzoneContainerWithTLS.java:325)
	at org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerWithTLS.testContainerDownload(TestOzoneContainerWithTLS.java:236)
...
Caused by: java.security.cert.CertificateExpiredException: NotAfter: Mon Dec 12 18:26:09 UTC 2022

• https://github.com/adoroszlai/ozone-build-results/blob/master/2022/12/12/19011/it-ozone/hadoop-ozone/integration-test/org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerWithTLS.txt
• https://github.com/adoroszlai/ozone-build-results/blob/master/2022/12/12/19014/it-ozone/hadoop-ozone/integration-test/org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerWithTLS.txt
• https://github.com/adoroszlai/ozone-build-results/blob/master/2022/12/13/19033/it-ozone/hadoop-ozone/integration-test/org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerWithTLS.txt

-------------------------------------------------------------------------------
Test set: org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerWithTLS
-------------------------------------------------------------------------------
Tests run: 6, Failures: 0, Errors: 1, Skipped: 2, Time elapsed: 21.974 s <<< FAILURE! - in org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerWithTLS
TestOzoneContainerWithTLS.testCreateOzoneContainer  Time elapsed: 8.529 s  <<< ERROR!
java.io.IOException: 
java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:436)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.lambda$sendCommandWithTraceIDAndRetry$0(XceiverClientGrpc.java:349)
	at org.apache.hadoop.hdds.tracing.TracingUtil.executeInSpan(TracingUtil.java:177)
	at org.apache.hadoop.hdds.tracing.TracingUtil.executeInNewSpan(TracingUtil.java:151)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:343)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:265)
	at org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerWithTLS.createContainer(TestOzoneContainerWithTLS.java:326)
	at org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerWithTLS.testCreateOzoneContainer(TestOzoneContainerWithTLS.java:189)
...
Caused by: java.security.cert.CertificateExpiredException: NotAfter: Thu Dec 15 07:43:25 UTC 2022

• https://github.com/adoroszlai/ozone-build-results/blob/master/2022/12/14/19063/it-ozone/hadoop-ozone/integration-test/org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerWithTLS.txt
• https://github.com/adoroszlai/ozone-build-results/blob/master/2022/12/15/19100/it-ozone/hadoop-ozone/integration-test/org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerWithTLS.txt

Issue Links

• is caused by: HDDS-7572 Use keyManager and trustManager provided by keyStoreFactory in datanode grpc services
• links to: GitHub Pull Request #4142
• links to: GitHub Pull Request #4151