Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-8937

bin/post (SimplePostTool) should stream/chunk if size not known

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.1
    • Component/s: SimplePostTool
    • Labels:
      None

      Description

      SimplePostTool.postData() takes a Long length arg that is sometimes null – for stdin and for processing some web pages for links. If it's null, we should this on the underlying HttpURLConnection:
      urlc.setChunkedStreamingMode(-1);. I have found that I can't use bin/post to stream large files from stdin since it the JDK's implementation wants to put the whole thing in memory first. Sadly this is true of curl as well. When I set chunk mode, it works. (FYI -1 tells it to use its default chunk size – 4k).

      FYI I'm using stdin because I've got massive JSON files zipped up.

        Activity

        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit f488dfac9cc516a05044279b5aa7e1254d15fa77 in lucene-solr's branch refs/heads/branch_6x from David Smiley
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f488dfa ]

        SOLR-8937: bin/post (SimplePostTool) should tell JDK to stream stdin instead of fully buffer
        (cherry picked from commit 037a403)

        Show
        jira-bot ASF subversion and git services added a comment - Commit f488dfac9cc516a05044279b5aa7e1254d15fa77 in lucene-solr's branch refs/heads/branch_6x from David Smiley [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f488dfa ] SOLR-8937 : bin/post (SimplePostTool) should tell JDK to stream stdin instead of fully buffer (cherry picked from commit 037a403)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 037a40316c897034c060041730ecefff4eca816a in lucene-solr's branch refs/heads/master from David Smiley
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=037a403 ]

        SOLR-8937: bin/post (SimplePostTool) should tell JDK to stream stdin instead of fully buffer

        Show
        jira-bot ASF subversion and git services added a comment - Commit 037a40316c897034c060041730ecefff4eca816a in lucene-solr's branch refs/heads/master from David Smiley [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=037a403 ] SOLR-8937 : bin/post (SimplePostTool) should tell JDK to stream stdin instead of fully buffer

          People

          • Assignee:
            dsmiley David Smiley
            Reporter:
            dsmiley David Smiley
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development