[METRON-1603] Fix multivalue field errors in Bro Solr schema

• Type: Bug
• Status: Done
• Priority: Major
• Resolution: Done
• Fix Version/s: 0.6.0
• Assignee: Michael Miklavcic
• Reporter: Michael Miklavcic
• Created: 05/Jun/18 19:41
• Updated: 21/May/19 20:18
• Resolved: 06/Jun/18 19:38

Description

Running some additional test data through Bro with multiValued fields revealed that the Solr schema for Bro needs some attention. Exceptions are thrown like the following for fields that may have many values but aren't declared as such in the Solr schema.

2018-06-05 07:34:11.903 o.a.s.d.executor Thread-6-indexingBolt-executor[3 3] [ERROR]
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://10.0.2.15:7574/solr/bro: ERROR: [doc=26643986-b4ce-4ffe-b84e-6fe45143ac16] multiple values encountered for non multiValued field answers: [www.cisco.com.akadns.net, origin-www.cisco.com, 2001:420:1201:2::a]
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:612) ~[stormjar.jar:?]
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279) ~[stormjar.jar:?]
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268) ~[stormjar.jar:?]
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:447) ~[stormjar.jar:?]
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:388) ~[stormjar.jar:?]
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1383) ~[stormjar.jar:?]
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1134) ~[stormjar.jar:?]
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073) ~[stormjar.jar:?]
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160) ~[stormjar.jar:?]
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106) ~[stormjar.jar:?]
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:71) ~[stormjar.jar:?]
at org.apache.metron.solr.writer.SolrWriter.write(SolrWriter.java:208) ~[stormjar.jar:?]
at org.apache.metron.writer.BulkWriterComponent.flush(BulkWriterComponent.java:239) [stormjar.jar:?]
at org.apache.metron.writer.BulkWriterComponent.write(BulkWriterComponent.java:217) [stormjar.jar:?]
at org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:258) [stormjar.jar:?]
at org.apache.storm.daemon.executor$fn__10252$tuple_action_fn__10254.invoke(executor.clj:735) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
at org.apache.storm.daemon.executor$mk_task_receiver$fn__10171.invoke(executor.clj:466) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
at org.apache.storm.disruptor$clojure_handler$reify__9685.onEvent(disruptor.clj:40) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
at org.apache.storm.daemon.executor$fn__10252$fn__10265$fn__10320.invoke(executor.clj:855) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]

Issue Links

• links to: GitHub Pull Request #1051