Nutch
  1. Nutch
  2. NUTCH-1026

Strip UTF-8 non-character codepoints

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: nutchgora
    • Fix Version/s: nutchgora
    • Component/s: indexer
    • Labels:
      None

      Description

      During a very large crawl i found a few documents producing non-character codepoints. When indexing to Solr this will yield the following exception:

      SEVERE: java.lang.RuntimeException: [was class java.io.CharConversionException] Invalid UTF-8 character 0xffff at char #1142033, byte #1155068)
              at com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18)
              at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731)
      

      Quite annoying! Here's quick fix for SolrWriter that'll pass the value of the content field to a method to strip away non-characters. I'm not too sure about this implementation but the tests i've done locally with a huge dataset now passes correctly. Here's a list of codepoints to strip away: http://unicode.org/cldr/utility/list-unicodeset.jsp?a=[:Noncharacter_Code_Point=True:]

      Please comment!

        Issue Links

          Activity

          Markus Jelsma created issue -
          Markus Jelsma made changes -
          Field Original Value New Value
          Link This issue supercedes NUTCH-1016 [ NUTCH-1016 ]
          Markus Jelsma made changes -
          Assignee Markus Jelsma [ markus17 ]
          Lewis John McGibbney made changes -
          Fix Version/s 2.1 [ 12321040 ]
          Fix Version/s nutchgora [ 12314893 ]
          Ferdy Galema made changes -
          Status Open [ 1 ] Closed [ 6 ]
          Fix Version/s nutchgora [ 12314893 ]
          Fix Version/s 2.1 [ 12321040 ]
          Resolution Fixed [ 1 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Markus Jelsma
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development