[NUTCH-2528] Getting error :-Invalid Utf-8 character while running a crawl job at indexer phase.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Auto Closed
• Affects Version/s: 2.3.1
• Fix Version/s: 2.5
• Component/s: indexer

Environment

OS- centos-release-6-9.el6.12.3.x86_64
Hadoop-2.5.2 cluster with 5 nodes
Nutch - 2.3.1
Hbase-0.98.8
Solr-5.4.1

• Flags: Patch, Important
• Assignee: Unassigned
• Reporter: Kshitij Shukla
• Created: 12/Mar/18 09:58
• Updated: 13/Oct/19 22:35
• Resolved: 13/Oct/19 22:35

Description

INFO [main] org.apache.hadoop.mapred.MapTask: Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector@4eb14055
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server ..... [com.ctc.wstx.exc.WstxLazyException] Invalid UTF-8 character 0xffff at char #20261885, byte #20308993)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:575)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:150)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:107)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:72)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:86)
at org.apache.nutch.indexwriter.solr.SolrIndexWriter.close(SolrIndexWriter.java:97)
at org.apache.nutch.indexer.IndexWriters.close(IndexWriters.java:114)
at org.apache.nutch.indexer.IndexerOutputFormat$1.close(IndexerOutputFormat.java:54)
at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1990)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:774)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:421)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)