Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-8937

bin/post (SimplePostTool) should stream/chunk if size not known

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.1
    • Component/s: SimplePostTool
    • Labels:
      None

      Description

      SimplePostTool.postData() takes a Long length arg that is sometimes null – for stdin and for processing some web pages for links. If it's null, we should this on the underlying HttpURLConnection:
      urlc.setChunkedStreamingMode(-1);. I have found that I can't use bin/post to stream large files from stdin since it the JDK's implementation wants to put the whole thing in memory first. Sadly this is true of curl as well. When I set chunk mode, it works. (FYI -1 tells it to use its default chunk size – 4k).

      FYI I'm using stdin because I've got massive JSON files zipped up.

        Attachments

          Activity

            People

            • Assignee:
              dsmiley David Smiley
              Reporter:
              dsmiley David Smiley
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: