Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24482

Not able to run simple SOLR application

    XMLWordPrintableJSON

    Details

    • Type: Brainstorming
    • Status: Resolved
    • Priority: Major
    • Resolution: Invalid
    • Affects Version/s: 1.6.0
    • Fix Version/s: None
    • Component/s: Spark Submit, YARN
    • Labels:
      None

      Description

      I am not able to run solr applciation may be due to "org.apache.httpcomponents" jars.

      I am using "org.apache.httpcomponents" 4.3.1 jars and my cluster has old jars:4.2.5.

      Event though I am excluding 4.2.5 jars , issue still exists. Adding exception at bottom. Any help is appreciated.

      =================================================

      spark2-submit \
      --master yarn \
      --deploy-mode client \
      --class com.citi.solrsearch.SolrSearch \
      --conf spark.executor.extraJavaOptions=-Djava.security.auth.login.config=/home/icttdnee/Sprinter/solrCitiscreening/jaas-client.conf \
      --conf spark.jars.excludes=/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p2964.3211/jars/httpclient-4.2.5.jar,/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p2964.3211/jars/httpcore-4.2.5.jar \
      --conf spark.driver.userClassPathFirst=false \
      --conf spark.executor.userClassPathFirst=false \
      --conf spark.driver.extraClassPath=/home/icttdnee/citiscreen_solr/httpclient-4.3.1.jar,/home/icttdnee/citiscreen_solr/httpcore-4.3.1.jar,/home/icttdnee/citiscreen_solr/httpclient-4.3.1.jar,/home/icttdnee/citiscreen_solr/httpcore-4.3.1.jar \
      --jars /home/icttdnee/citiscreen_solr/spark-solr-2.1.0.jar,/home/pd94468/scala-logging_2.11-3.4.0.jar,/home/icttdnee/citiscreen_solr/spark-core_2.11-1.6.0.jar,/home/icttdnee/citiscreen_solr/httpclient-4.3.1.jar,/home/icttdnee/citiscreen_solr/httpcore-4.3.1.jar \
      /home/icttdnee/citiscreen_solr/spark-solr-1.0-SNAPSHOT-jar-with-dependencies.jar \
      https://bdgtr012x07h2.nam.nsroot.net:8985/solr/icttdnee_ttsd_collection \
      &q=alert_id:[16000%20TO%20*]

      =================================================

      Maven Snippet:

      <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.10</artifactId>
      <version>2.0.2</version>
      </dependency>

      <!-- https://mvnrepository.com/artifact/com.lucidworks.spark/spark-solr -->
      <dependency>
      <groupId>com.lucidworks.spark</groupId>
      <artifactId>spark-solr</artifactId>
      <version>3.5.0</version>
      </dependency>

      <!-- https://mvnrepository.com/artifact/org.apache.solr/solr-core -->
      <dependency>
      <groupId>org.apache.solr</groupId>
      <artifactId>solr-core</artifactId>
      <version>4.10.3</version>
      </dependency>
      <!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient -->
      <dependency>
      <groupId>org.apache.httpcomponents</groupId>
      <artifactId>httpclient</artifactId>
      <version>4.3.1</version>
      </dependency>
      <!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpcore -->
      <dependency>
      <groupId>org.apache.httpcomponents</groupId>
      <artifactId>httpcore</artifactId>
      <version>4.3.1</version>
      </dependency>

      ==============================================================

       

       

      18/06/07 05:08:00 INFO spark.SparkContext: Registered listener com.pepperdata.spark.metrics.PepperdataSparkListener
      Exception in thread "main" com.google.common.util.concurrent.ExecutionError: java.lang.NoSuchFieldError: INSTANCE
      at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2232)
      at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
      at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969)
      at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829)
      at com.lucidworks.spark.util.SolrSupport$.getCachedCloudClient(SolrSupport.scala:245)
      at com.lucidworks.spark.util.SolrQuerySupport$.getUniqueKey(SolrQuerySupport.scala:91)
      at com.lucidworks.spark.rdd.SolrRDD.<init>(SolrRDD.scala:40)
      at com.lucidworks.spark.rdd.SelectSolrRDD.<init>(SelectSolrRDD.scala:30)
      at com.lucidworks.spark.rdd.SelectSolrRDD$.apply(SelectSolrRDD.scala:168)
      at com.lucidworks.spark.rdd.SolrJavaRDD.get(SolrJavaRDD.java:48)
      at com.citi.solrsearch.SolrSearch.main(SolrSearch.java:48)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      Caused by: java.lang.NoSuchFieldError: INSTANCE
      at org.apache.http.impl.conn.HttpClientConnectionOperator.<init>(HttpClientConnectionOperator.java:75)
      at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:149)
      at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:136)
      at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:112)
      at org.apache.solr.client.solrj.impl.HttpClientUtil.createPoolingConnectionManager(HttpClientUtil.java:228)
      at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:223)
      at com.lucidworks.spark.util.SolrSupport$.getSolrCloudClient(SolrSupport.scala:218)
      at com.lucidworks.spark.util.SolrSupport$.getNewSolrCloudClient(SolrSupport.scala:241)
      at com.lucidworks.spark.util.CacheCloudSolrClient$$anon$1.load(SolrSupport.scala:39)
      at com.lucidworks.spark.util.CacheCloudSolrClient$$anon$1.load(SolrSupport.scala:37)
      at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
      at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
      at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
      at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              pravinda16 Pravin Dahare
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: