[NUTCH-2715] WARCExporter fails on large records

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.15
• Fix Version/s: 1.16
• Assignee: Unassigned
• Reporter: Yossi Tamari
• Created: 06/May/19 12:20
• Updated: 28/Jan/21 13:55
• Resolved: 24/May/19 13:28

Description

com.martinkl.warc.WARCRecord throws an IllegalStateException when a single line is over 10,000 bytes. Since this exception is not caught in WARCExporter, it fails the whole export.

I doubt that validity of the limitation in WARCRecord, but regardless, I think WARCExporter should catch the exception and skip to the next record.

(See also https://github.com/ept/warc-hadoop/issues/5)