Nutch
  1. Nutch
  2. NUTCH-1308

Unnecessary truncate content configuration, and logging in parse-zip/ZipParser

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.4, nutchgora
    • Fix Version/s: 2.4
    • Component/s: parser
    • Labels:
      None

      Description

      Two issues here...

      1) Recently ferdy committed NUTCH-965 which skips parsing of truncated documents. Parse zip has it's own implementation for the same when it should really draw on the aforementioned implementation.
      2) If (in the offending piece of code mentioned above) truncation occurs, we get an incorrect log message the "Parser can't handle incomplete pdf files"!!! This is incorrect, shouldn't be there, and should be removed.

      
      72      if (contentLen != null && contentInBytes.length != len) {
      73 	return new ParseStatus(ParseStatus.FAILED,
      74 	ParseStatus.FAILED_TRUNCATED, "Content truncated at "
      75 	+ contentInBytes.length
      76 	+ " bytes. Parser can't handle incomplete pdf file.")
      77 	.getEmptyParseResult(content.getUrl(), getConf());
      78 	}
      

      For clarity, the issue is present in both Nutchgora branch[1] and Nutch trunk[2]

      [1] https://svn.apache.org/viewvc/nutch/branches/nutchgora/src/plugin/parse-zip/src/java/org/apache/nutch/parse/zip/ZipParser.java?diff_format=h&view=markup
      [2] https://svn.apache.org/viewvc/nutch/trunk/src/plugin/parse-zip/src/java/org/apache/nutch/parse/zip/ZipParser.java?diff_format=h&view=markup
      [2]

        Issue Links

          Activity

          Lewis John McGibbney made changes -
          Link This issue is duplicated by NUTCH-1603 [ NUTCH-1603 ]
          Lewis John McGibbney made changes -
          Fix Version/s 1.8 [ 12324326 ]
          Fix Version/s 2.4 [ 12324540 ]
          Fix Version/s 2.3 [ 12324325 ]
          Sebastian Nagel made changes -
          Fix Version/s 1.8 [ 12324326 ]
          Lewis John McGibbney made changes -
          Fix Version/s 2.3 [ 12324325 ]
          Fix Version/s 1.7 [ 12323281 ]
          Fix Version/s 2.2 [ 12323285 ]
          Hide
          Lewis John McGibbney added a comment -

          This issue should also incorporate a main class to ZipParser which would enable adhoc parsing of files located on a local file system.

          Show
          Lewis John McGibbney added a comment - This issue should also incorporate a main class to ZipParser which would enable adhoc parsing of files located on a local file system.
          Lewis John McGibbney made changes -
          Fix Version/s 2.2 [ 12323285 ]
          Lewis John McGibbney made changes -
          Assignee Lewis John McGibbney [ lewismc ]
          Lewis John McGibbney made changes -
          Fix Version/s 1.7 [ 12323281 ]
          Fix Version/s 1.6 [ 12319941 ]
          Markus Jelsma made changes -
          Field Original Value New Value
          Fix Version/s 1.6 [ 12319941 ]
          Fix Version/s nutchgora [ 12314893 ]
          Fix Version/s 1.5 [ 12318246 ]
          Hide
          Markus Jelsma added a comment -

          20120304-push-1.6

          Show
          Markus Jelsma added a comment - 20120304-push-1.6
          Lewis John McGibbney created issue -

            People

            • Assignee:
              Lewis John McGibbney
              Reporter:
              Lewis John McGibbney
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:

                Development