[SOLR-5101] Invalid UTF-8 character 0xfffe during shard update

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 4.3
• Component/s: clients - java

Environment

Ubuntu 12.04.2
java version "1.6.0_27"
OpenJDK Runtime Environment (IcedTea6 1.12.5) (6b27-1.12.5-0ubuntu0.12.04.1)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

• Assignee: Unassigned
• Reporter: Federico Chiacchiaretta
• Created: 01/Aug/13 08:16
• Updated: 13/Aug/13 14:42
• Resolved: 01/Aug/13 12:06

Description

On data import from a PostgreSQL db, I get the following error in solr.log:

ERROR - 2013-08-01 09:51:00.217; org.apache.solr.common.SolrException; shard update error RetryNode: http://172.16.201.173:8983/solr/archive/:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Invalid UTF-8 character 0xfffe at char #416, byte #127)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:402)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:332)
at org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:306)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)

This prevents the document from being successfully added to the index, and a few documents targeting the same shard are also missing.
This happens silently, because data import completes successfully, and the whole number of documents reported as Added includes those who failed (and are actually lost).

Is there a known workaround for this issue?

Regards,
Federico Chiacchiaretta