[SOLR-7948] MapReduceIndexerTool of solr 5.2.1 doesn't work with hadoop 2.7.1

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 5.2.1
• Component/s: contrib - MapReduce

Environment

OS:suse 11
JDK:java version "1.7.0_65"
Java(TM) SE Runtime Environment (build 1.7.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
HADOOP:hadoop 2.7.1
SOLR:5.2.1

• Assignee: Mark Miller
• Reporter: davidchiu
• Created: 20/Aug/15 03:32
• Updated: 24/Mar/17 16:54
• Resolved: 24/Mar/17 16:54

Description

When I used MapReduceIndexerTool of 5.2.1 to index files, I got follwoing errors,but I used 4.9.0's MapReduceIndexerTool, it did work with hadoop 2.7.1.
Exception ERROR as following:
INFO - 2015-08-20 11:44:45.155; [ ] org.apache.solr.hadoop.HeartBeater; Heart beat reporting class is org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
INFO - 2015-08-20 11:44:45.161; [ ] org.apache.solr.hadoop.SolrRecordWriter; Using this unpacked directory as solr home: /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip
INFO - 2015-08-20 11:44:45.162; [ ] org.apache.solr.hadoop.SolrRecordWriter; Creating embedded Solr server with solrHomeDir: /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip, fs: DFS[DFSClient[clientName=DFSClient_attempt_1440040092614_0004_r_000001_0_1678264055_1, ugi=root (auth:SIMPLE)]], outputShardDir: hdfs://127.0.0.1:9000/tmp/outdir/reducers/_temporary/1/_temporary/attempt_1440040092614_0004_r_000001_0/part-r-00001
INFO - 2015-08-20 11:44:45.194; [ ] org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader for directory: '/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/'
INFO - 2015-08-20 11:44:45.206; [ ] org.apache.solr.hadoop.HeartBeater; HeartBeat thread running
INFO - 2015-08-20 11:44:45.207; [ ] org.apache.solr.hadoop.HeartBeater; Issuing heart beat for 1 threads
INFO - 2015-08-20 11:44:45.418; [ ] org.apache.solr.hadoop.SolrRecordWriter; Constructed instance information solr.home /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip (/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip), instance dir /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/, conf dir /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/conf/, writing index to solr.data.dir hdfs://127.0.0.1:9000/tmp/outdir/reducers/_temporary/1/_temporary/attempt_1440040092614_0004_r_000001_0/part-r-00001/data, with permdir hdfs://127.0.0.10:9000/tmp/outdir/reducers/_temporary/1/_temporary/attempt_1440040092614_0004_r_000001_0/part-r-00001
INFO - 2015-08-20 11:44:45.426; [ ] org.apache.solr.core.SolrXmlConfig; Loading container configuration from /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/solr.xml
INFO - 2015-08-20 11:44:45.474; [ ] org.apache.solr.core.CorePropertiesLocator; Config-defined core root directory: /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip
INFO - 2015-08-20 11:44:45.503; [ ] org.apache.solr.core.CoreContainer; New CoreContainer 1656436773
INFO - 2015-08-20 11:44:45.503; [ ] org.apache.solr.core.CoreContainer; Loading cores into CoreContainer [instanceDir=/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/]
INFO - 2015-08-20 11:44:45.503; [ ] org.apache.solr.core.CoreContainer; loading shared library: /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/lib
WARN - 2015-08-20 11:44:45.504; [ ] org.apache.solr.core.SolrResourceLoader; Can't find (or read) directory to add to classloader: lib (resolved as: /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/lib).
INFO - 2015-08-20 11:44:45.520; [ ] org.apache.solr.handler.component.HttpShardHandlerFactory; created with socketTimeout : 600000,connTimeout : 60000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
FATAL - 2015-08-20 11:44:45.526; [ ] org.apache.hadoop.mapred.YarnChild; Error running child : java.lang.VerifyError: Bad return type
Exception Details:
Location:
org/apache/solr/client/solrj/impl/HttpClientUtil.createClient(Lorg/apache/solr/common/params/SolrParams;Lorg/apache/http/conn/ClientConnectionManager;)Lorg/apache/http/impl/client/CloseableHttpClient; @62: areturn
Reason:
Type 'org/apache/http/impl/client/DefaultHttpClient' (current frame, stack[0]) is not assignable to 'org/apache/http/impl/client/CloseableHttpClient' (from method signature)
Current Frame:
bci: @62
flags: { }
locals:

{ 'org/apache/solr/common/params/SolrParams', 'org/apache/http/conn/ClientConnectionManager', 'org/apache/solr/common/params/ModifiableSolrParams', 'org/apache/http/impl/client/DefaultHttpClient' }

stack:

{ 'org/apache/http/impl/client/DefaultHttpClient' }

Bytecode:
0000000: bb00 0359 2ab7 0004 4db2 0005 b900 0601
0000010: 0099 001e b200 05bb 0007 59b7 0008 1209
0000020: b600 0a2c b600 0bb6 000c b900 0d02 00bb
0000030: 0011 592b b700 124e 2d2c b800 102d b0
Stackmap Table:
append_frame(@47,Object127)

at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:166)
at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:49)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:328)
at org.apache.solr.hadoop.SolrRecordWriter.createEmbeddedSolrServer(SolrRecordWriter.java:163)
at org.apache.solr.hadoop.SolrRecordWriter.<init>(SolrRecordWriter.java:119)
at org.apache.solr.hadoop.SolrOutputFormat.getRecordWriter(SolrOutputFormat.java:163)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:540)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:614)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Issue Links

• is duplicated by: SOLR-7811 mapreduce contrib has an issue with morphlines lib relying on solr code from a standard release leading to runtime class mismatch errors.
• is superceded by: SOLR-9221 Remove Solr contribs: map-reduce, morphlines-core and morphlines-cell