Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-2716

protocol-http: Response headers are not stored for a compressed response

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.15
    • Fix Version/s: 1.16
    • Component/s: protocol
    • Labels:
      None

      Description

      Even when store.http.headers=true, the HTTP headers are not saved for a gzipped or deflated response, because they may contain an incorrect content-length header.

      This causes WARCExporter to generate "resource" (headerless) entries instead of "response" entries.

      While I can see why reporting the wrong content-encoding and length may be a bug, removing all the headers is not a fix.

      I am not submitting a patch yet since I'm not sure what the best fix is, but I guess the best patch is to remove those two header lines and store the rest of the headers. If there is no objection, I can submit a patch that does this. Otherwise, what would be a better fix?

        Attachments

          Activity

            People

            • Assignee:
              snagel Sebastian Nagel
              Reporter:
              yossi Yossi Tamari
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: