[SOLR-8937] bin/post (SimplePostTool) should stream/chunk if size not known

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 6.1
• Component/s: SimplePostTool
• Assignee: David Smiley
• Reporter: David Smiley
• Created: 04/Apr/16 04:49
• Updated: 17/Jun/16 13:11
• Resolved: 14/Apr/16 17:53

Description

SimplePostTool.postData() takes a Long length arg that is sometimes null – for stdin and for processing some web pages for links. If it's null, we should this on the underlying HttpURLConnection:
urlc.setChunkedStreamingMode(-1);. I have found that I can't use bin/post to stream large files from stdin since it the JDK's implementation wants to put the whole thing in memory first. Sadly this is true of curl as well. When I set chunk mode, it works. (FYI -1 tells it to use its default chunk size – 4k).

FYI I'm using stdin because I've got massive JSON files zipped up.