Nutch
  1. Nutch
  2. NUTCH-1016

Strip UTF-8 non-character codepoints

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.3
    • Fix Version/s: 1.4
    • Component/s: indexer
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      During a very large crawl i found a few documents producing non-character codepoints. When indexing to Solr this will yield the following exception:

      SEVERE: java.lang.RuntimeException: [was class java.io.CharConversionException] Invalid UTF-8 character 0xffff at char #1142033, byte #1155068)
              at com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18)
              at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731)
      

      Quite annoying! Here's quick fix for SolrWriter that'll pass the value of the content field to a method to strip away non-characters. I'm not too sure about this implementation but the tests i've done locally with a huge dataset now passes correctly. Here's a list of codepoints to strip away: http://unicode.org/cldr/utility/list-unicodeset.jsp?a=[:Noncharacter_Code_Point=True:]

      Please comment!

      1. NUTCH-1016-2.0.patch
        3 kB
        Markus Jelsma
      2. NUTCH-1016-1.4-4.patch
        3 kB
        Markus Jelsma

        Issue Links

          Activity

          Markus Jelsma created issue -
          Markus Jelsma made changes -
          Field Original Value New Value
          Attachment NUTCH-1016-1.4.patch [ 12483964 ]
          Markus Jelsma made changes -
          Attachment NUTCH-1016-1.4.patch [ 12483964 ]
          Markus Jelsma made changes -
          Attachment NUTCH-1016-1.4-2.patch [ 12483966 ]
          Markus Jelsma made changes -
          Attachment NUTCH-1016-1.4-2.patch [ 12483966 ]
          Markus Jelsma made changes -
          Attachment NUTCH-1016-1.4-3.patch [ 12484441 ]
          Markus Jelsma made changes -
          Attachment NUTCH-1016-1.4-3.patch [ 12484441 ]
          Markus Jelsma made changes -
          Attachment NUTCH-1016-1.4-4.patch [ 12484442 ]
          Markus Jelsma made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Markus Jelsma made changes -
          Attachment NUTCH-1016-2.0.patch [ 12484764 ]
          Markus Jelsma made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Markus Jelsma made changes -
          Fix Version/s 1.4 [ 12316519 ]
          Markus Jelsma made changes -
          Fix Version/s 1.4 [ 12316519 ]
          Markus Jelsma made changes -
          Link This issue is superceded by NUTCH-1026 [ NUTCH-1026 ]
          Markus Jelsma made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Fix Version/s 2.0 [ 12314893 ]
          Resolution Fixed [ 1 ]
          Markus Jelsma made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Markus Jelsma
              Reporter:
              Markus Jelsma
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development