[HDDS-5620] Fix and extend hadoop.http.idle_timeout.ms to 60k

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.2.0
• Component/s: S3
• Labels: pull-request-available
• External issue ID: HADOOP-15696
• Assignee: UENISHI Kota
• Reporter: UENISHI Kota
• Created: 17/Aug/21 02:22
• Updated: 19/Aug/21 04:27
• Resolved: 19/Aug/21 04:27

Description

The default value of hadoop.http.idle_timeout.ms seems to be inteneded as 10k in HttpServer2.java, but it is ignored and actually falls into 1000 defined in core-site.xml in hadoop-common implemented in HADOOP-15696 . I would also propose extending the default value to 60k, which is commonly used value among other Hadoop projects.

This leads to an error in S3G when connected by a slow client, like:

2021-08-16 19:55:41,329 [qtp1516500233-135] WARN org.eclipse.jetty.server.HttpChannel: handleException /ksugihara.sandbox/kubernetes11/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/grpctrace.go java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 1000/1000 ms
2021-08-16 19:55:41,329 [qtp1516500233-135] WARNorg.eclipse.jetty.server.HttpChannelState: unhandled due to prior sendError
javax.servlet.ServletException: javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.io.IOException: java.util.concurrent.TimeoutException: idle timeout expired: 1000/1000 ms
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
(snip)
Caused by: javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired:
1000/1000 ms
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
(snip)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
... 40 more
Caused by: java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 1000/1000 ms
at org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1124)
at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:333)
at java.io.InputStream.read(InputStream.java:101)
at org.glassfish.jersey.message.internal.EntityInputStream.read(EntityInputStream.java:98)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2146)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:2102)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2123)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:2078)
at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.put(ObjectEndpoint.java:200)
at sun.reflect.GeneratedMethodAccessor65.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)

Issue Links

• links to: GitHub Pull Request #2540