Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-7948

MapReduceIndexerTool of solr 5.2.1 doesn't work with hadoop 2.7.1

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 5.2.1
    • Fix Version/s: None
    • Component/s: contrib - MapReduce
    • Labels:
      None
    • Environment:

      OS:suse 11
      JDK:java version "1.7.0_65"
      Java(TM) SE Runtime Environment (build 1.7.0_65-b17)
      Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
      HADOOP:hadoop 2.7.1
      SOLR:5.2.1

      Description

      When I used MapReduceIndexerTool of 5.2.1 to index files, I got follwoing errors,but I used 4.9.0's MapReduceIndexerTool, it did work with hadoop 2.7.1.
      Exception ERROR as following:
      INFO - 2015-08-20 11:44:45.155; [ ] org.apache.solr.hadoop.HeartBeater; Heart beat reporting class is org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
      INFO - 2015-08-20 11:44:45.161; [ ] org.apache.solr.hadoop.SolrRecordWriter; Using this unpacked directory as solr home: /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip
      INFO - 2015-08-20 11:44:45.162; [ ] org.apache.solr.hadoop.SolrRecordWriter; Creating embedded Solr server with solrHomeDir: /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip, fs: DFS[DFSClient[clientName=DFSClient_attempt_1440040092614_0004_r_000001_0_1678264055_1, ugi=root (auth:SIMPLE)]], outputShardDir: hdfs://127.0.0.1:9000/tmp/outdir/reducers/_temporary/1/_temporary/attempt_1440040092614_0004_r_000001_0/part-r-00001
      INFO - 2015-08-20 11:44:45.194; [ ] org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader for directory: '/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/'
      INFO - 2015-08-20 11:44:45.206; [ ] org.apache.solr.hadoop.HeartBeater; HeartBeat thread running
      INFO - 2015-08-20 11:44:45.207; [ ] org.apache.solr.hadoop.HeartBeater; Issuing heart beat for 1 threads
      INFO - 2015-08-20 11:44:45.418; [ ] org.apache.solr.hadoop.SolrRecordWriter; Constructed instance information solr.home /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip (/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip), instance dir /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/, conf dir /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/conf/, writing index to solr.data.dir hdfs://127.0.0.1:9000/tmp/outdir/reducers/_temporary/1/_temporary/attempt_1440040092614_0004_r_000001_0/part-r-00001/data, with permdir hdfs://127.0.0.10:9000/tmp/outdir/reducers/_temporary/1/_temporary/attempt_1440040092614_0004_r_000001_0/part-r-00001
      INFO - 2015-08-20 11:44:45.426; [ ] org.apache.solr.core.SolrXmlConfig; Loading container configuration from /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/solr.xml
      INFO - 2015-08-20 11:44:45.474; [ ] org.apache.solr.core.CorePropertiesLocator; Config-defined core root directory: /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip
      INFO - 2015-08-20 11:44:45.503; [ ] org.apache.solr.core.CoreContainer; New CoreContainer 1656436773
      INFO - 2015-08-20 11:44:45.503; [ ] org.apache.solr.core.CoreContainer; Loading cores into CoreContainer [instanceDir=/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/]
      INFO - 2015-08-20 11:44:45.503; [ ] org.apache.solr.core.CoreContainer; loading shared library: /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/lib
      WARN - 2015-08-20 11:44:45.504; [ ] org.apache.solr.core.SolrResourceLoader; Can't find (or read) directory to add to classloader: lib (resolved as: /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/lib).
      INFO - 2015-08-20 11:44:45.520; [ ] org.apache.solr.handler.component.HttpShardHandlerFactory; created with socketTimeout : 600000,connTimeout : 60000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
      FATAL - 2015-08-20 11:44:45.526; [ ] org.apache.hadoop.mapred.YarnChild; Error running child : java.lang.VerifyError: Bad return type
      Exception Details:
      Location:
      org/apache/solr/client/solrj/impl/HttpClientUtil.createClient(Lorg/apache/solr/common/params/SolrParams;Lorg/apache/http/conn/ClientConnectionManager;)Lorg/apache/http/impl/client/CloseableHttpClient; @62: areturn
      Reason:
      Type 'org/apache/http/impl/client/DefaultHttpClient' (current frame, stack[0]) is not assignable to 'org/apache/http/impl/client/CloseableHttpClient' (from method signature)
      Current Frame:
      bci: @62
      flags: { }
      locals:

      { 'org/apache/solr/common/params/SolrParams', 'org/apache/http/conn/ClientConnectionManager', 'org/apache/solr/common/params/ModifiableSolrParams', 'org/apache/http/impl/client/DefaultHttpClient' }

      stack:

      { 'org/apache/http/impl/client/DefaultHttpClient' }

      Bytecode:
      0000000: bb00 0359 2ab7 0004 4db2 0005 b900 0601
      0000010: 0099 001e b200 05bb 0007 59b7 0008 1209
      0000020: b600 0a2c b600 0bb6 000c b900 0d02 00bb
      0000030: 0011 592b b700 124e 2d2c b800 102d b0
      Stackmap Table:
      append_frame(@47,Object127)

      at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:166)
      at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:49)
      at org.apache.solr.core.CoreContainer.load(CoreContainer.java:328)
      at org.apache.solr.hadoop.SolrRecordWriter.createEmbeddedSolrServer(SolrRecordWriter.java:163)
      at org.apache.solr.hadoop.SolrRecordWriter.<init>(SolrRecordWriter.java:119)
      at org.apache.solr.hadoop.SolrOutputFormat.getRecordWriter(SolrOutputFormat.java:163)
      at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:540)
      at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:614)
      at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
      at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:415)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
      at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                markrmiller@gmail.com Mark Miller
                Reporter:
                davidchiu davidchiu
              • Votes:
                1 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: