[HDDS-11257] Ozone write does not work when http proxy is set for the JVM.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Labels: pull-request-available
• Assignee: Sadanand Shenoy
• Reporter: Sadanand Shenoy
• Created: 31/Jul/24 13:10
• Updated: 07/Aug/24 08:49
• Resolved: 07/Aug/24 08:49

Description

Caused by: java.util.concurrent.CompletionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNKNOWN: Channel Pipeline: [ProtocolNegotiators$ProxyProtocolNegotiationHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
    at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
    at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
    at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607)
    at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
    ... 16 more
Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNKNOWN: Channel Pipeline: [ProtocolNegotiators$ProxyProtocolNegotiationHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
    at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:539)
    ... 13 more
Caused by: java.lang.NoClassDefFoundError: org/apache/ratis/thirdparty/io/netty/handler/proxy/HttpProxyHandler
    at org.apache.ratis.thirdparty.io.grpc.netty.ProtocolNegotiators$ProxyProtocolNegotiationHandler.protocolNegotiationEventTriggered(ProtocolNegotiators.java:526)
    at org.apache.ratis.thirdparty.io.grpc.netty.ProtocolNegotiators$ProtocolNegotiationHandler.userEventTriggered(ProtocolNegotiators.java:1061)
    at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:398)
    at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376)
    at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:368)
    at org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1428)
    at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:396)
    at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376)
    at org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:913)
    at org.apache.ratis.thirdparty.io.grpc.netty.WriteBufferingAndExceptionHandler.handlerAdded(WriteBufferingAndExceptionHandler.java:62)
    at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:1114)
    at org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)
    at org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46)
    at org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463)
    at org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115)
    at org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650)
    at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:514)
    at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429)
    at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486)
    at org.apache.ratis.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
    at org.apache.ratis.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
    at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
    at org.apache.ratis.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413)
    at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at org.apache.ratis.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    ... 1 more
24/07/31 10:57:31 WARN io.KeyOutputStream: Failure for replica index: 4, DatanodeDetails: f885d6f6-b33c-4bf2-a482-ffbfb001c0a 

When an http proxy is configured for the JVM, there are failures in leader election due to missing dependencies in the Ratis thirdparty jars.

To reproduce on a docker-env , set the following and try an ozone key write.

export OZONE_CLIENT_OPTS="-Dhttps.proxyHost=10.20.30.40 -Dhttps.proxyPort=552  $OZONE_CLIENT_OPTS” 

To reproduce via integration tests use the surefire plugin to set the jvm property and run an integration test.

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
  <argLine>-Dhttps.proxyHost=10.20.30.40 -Dhttps.proxyPort=552</argLine>
</configuration>
</plugin> 

This might not need an ozone fix but since the problem is encountered in Ozone , I created this jira.

Issue Links

• is caused by: RATIS-2133 Ignore http proxy for ratis grpc client & server
• links to: GitHub Pull Request #7036