Details
-
New Feature
-
Status: Open
-
Major
-
Resolution: Unresolved
-
2.5.0-beta1
-
None
Description
In enterprise gadget development, there are requirements to allow files upload/download up to 512MB. Shindig server is processing the fetched response from target url every time, convert the stream fetched into byte array and store the byte array in HttpResponse, later on the HttpResponse can be copied to a ServletResponse and return to client. This solution has severe problem when the stream is huge: it could easily cause the JVM to exhaust its heap under moderate to heavy load.
Dan's patch about proxied-form-post would solve the upload. https://issues.apache.org/jira/browse/SHINDIG-1695
We'd like to propose a download solution: streamed request. The idea is to add a new field in gadgets.io.RequestParameters.STREAMED, and update gadgets.io.getProxyUrl() to consider the case that this is a streamed request, and return a proxied URL like this: http://localhost:8080/gadgets/proxy?container=default&refresh=0&url=http%3A%2F%2Flocalhost%3A8080%2Ftest%2Ftest.rar&streamed=true
Then Shindig server side ProxyHandler would be able to tell whether the response stream need to be processed in Shindig server or not. If it is a streamed request, Shindig server won't process it, just set the stream into HttpResponse and return it to ServletReponse. All the other processes such as response headers, security, caching etc will be kept the same as they were.
The changes need to be made in Shindig server are Proxy URL handling, HttpResponse, HttpResponseBuilder, ProxyHandler, ProxyServlet, DefaultRequestPipeline and BasicHttpFetcher.
In our project environment, we've injected a our own HttpFetcher, right now, our prototype already work well with those additional changes mentioned above in Shindig. We would like to create a patch for community to review.
The patch will be available for review in 2 week, in the meantime, we would like to get community's comments about this approach.