Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-10981

Allow update to load gzip files

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 6.6
    • Fix Version/s: 7.6, 8.0
    • Component/s: SolrJ
    • Labels:

      Description

      We currently import large CSV files. We store them in gzip files as they compress at around 80%.

      To import them we must gunzip them and then import them. After that we no longer need the decompressed files.

      This patch allows directly opening either URL, or local files that are gzipped.

      For URLs, to determine if the file is gzipped, it will check the content encoding=="gzip" or if the file ends in ".gz"

      For files, if the file ends in ".gz" then it will assume the file is gzipped.

      I have tested the patch with 4.10.4, 6.6.0, 7.0.1 and master from git.

        Attachments

        1. SOLR-10981.patch
          18 kB
          Andrew Lundgren
        2. SOLR-10981.patch
          18 kB
          Andrew Lundgren
        3. SOLR-10981.patch
          18 kB
          Andrew Lundgren
        4. SOLR-10981.patch
          16 kB
          Andrew Lundgren
        5. SOLR-10981.patch
          11 kB
          Andrew Lundgren
        6. SOLR-10981.patch
          9 kB
          Andrew Lundgren

        Issue Links

          Activity

            People

            • Assignee:
              dsmiley David Smiley
              Reporter:
              lundgren Andrew Lundgren

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment