Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-2716

protocol-http: Response headers are not stored for a compressed response

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.15
    • 1.16
    • protocol
    • None

    Description

      Even when store.http.headers=true, the HTTP headers are not saved for a gzipped or deflated response, because they may contain an incorrect content-length header.

      This causes WARCExporter to generate "resource" (headerless) entries instead of "response" entries.

      While I can see why reporting the wrong content-encoding and length may be a bug, removing all the headers is not a fix.

      I am not submitting a patch yet since I'm not sure what the best fix is, but I guess the best patch is to remove those two header lines and store the rest of the headers. If there is no objection, I can submit a patch that does this. Otherwise, what would be a better fix?

      Attachments

        Activity

          People

            snagel Sebastian Nagel
            yossi Yossi Tamari
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: